You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2013/12/29 01:25:31 UTC
[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 2242 - Still
Failing
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/2242/
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch
Error Message:
There were expected update fails expected:<0> but was:<1>
Stack Trace:
java.lang.AssertionError: There were expected update fails expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([262A32FED6183C06:A7CCBCE6A1475C3A]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:202)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:849)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:679)
Build Log:
[...truncated 10274 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
[junit4] 2> 1665814 T3204 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 1665820 T3204 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeyNothingIsSafeTest-1388276348573
[junit4] 2> 1665821 T3204 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1665822 T3205 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1665923 T3204 oasc.ZkTestServer.run start zk server on port:42233
[junit4] 2> 1665925 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1665930 T3212 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56261adf name:ZooKeeperConnection Watcher:127.0.0.1:42233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1665930 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1665931 T3204 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1665981 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1665983 T3215 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66736dcc name:ZooKeeperConnection Watcher:127.0.0.1:42233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1665984 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1665984 T3204 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1665993 T3204 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1665996 T3204 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1665999 T3204 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1666007 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1666008 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1666021 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 1666022 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1666127 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1666128 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1666137 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1666138 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1666142 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1666143 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1666146 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1666147 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1666150 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1666151 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 1666155 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1666156 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1666160 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1666160 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1666164 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1666164 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1666168 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1666169 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1666467 T3204 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1666471 T3204 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:28670
[junit4] 2> 1666471 T3204 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1666472 T3204 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1666472 T3204 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925
[junit4] 2> 1666473 T3204 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925/'
[junit4] 2> 1666513 T3204 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925/solr.xml
[junit4] 2> 1666594 T3204 oasc.CoreContainer.<init> New CoreContainer 1445083937
[junit4] 2> 1666595 T3204 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925/]
[junit4] 2> 1666597 T3204 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1666598 T3204 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1666598 T3204 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1666599 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1666599 T3204 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1666599 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1666600 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1666600 T3204 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1666601 T3204 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1666617 T3204 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1666618 T3204 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1666619 T3204 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1666619 T3204 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42233/solr
[junit4] 2> 1666620 T3204 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1666621 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1666624 T3227 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60899459 name:ZooKeeperConnection Watcher:127.0.0.1:42233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1666624 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1666627 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1666636 T3230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e6f0d5e name:ZooKeeperConnection Watcher:127.0.0.1:42233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1666636 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1666638 T3204 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1666649 T3204 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1666653 T3204 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1666655 T3204 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:28670_
[junit4] 2> 1666662 T3204 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:28670_
[junit4] 2> 1666666 T3204 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1666680 T3204 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1666683 T3204 oasc.Overseer.start Overseer (id=90982078780538883-127.0.0.1:28670_-n_0000000000) starting
[junit4] 2> 1666694 T3204 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1666705 T3232 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1666706 T3204 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1666709 T3204 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1666711 T3204 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1666714 T3231 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1666720 T3233 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1666720 T3233 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1666721 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1666721 T3233 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1666723 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1666724 T3231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:28670",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:28670_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1666724 T3231 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 1666724 T3231 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1666727 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1666737 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1667723 T3233 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1667723 T3233 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925/collection1
[junit4] 2> 1667724 T3233 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1667725 T3233 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1667725 T3233 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1667726 T3233 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925/collection1/'
[junit4] 2> 1667728 T3233 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925/collection1/lib/classes/' to classloader
[junit4] 2> 1667729 T3233 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925/collection1/lib/README' to classloader
[junit4] 2> 1667796 T3233 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 1667846 T3 oasu.ConcurrentLRUCache.finalize ERROR ConcurrentLRUCache was not destroyed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
[junit4] 2> 1667871 T3233 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1667972 T3233 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1667995 T3233 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1668541 T3233 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1668545 T3233 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1668547 T3233 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1668553 T3233 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1668578 T3233 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1668621 T3233 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/control/data/
[junit4] 2> 1668621 T3233 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4e14093f
[junit4] 2> 1668623 T3233 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/control/data
[junit4] 2> 1668624 T3233 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/control/data/index/
[junit4] 2> 1668625 T3233 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 1668626 T3233 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/control/data/index
[junit4] 2> 1668627 T3233 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=316730711, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 1668657 T3233 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 1668657 T3233 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1668660 T3233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1668661 T3233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1668661 T3233 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1668661 T3233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1668662 T3233 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1668662 T3233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1668662 T3233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1668663 T3233 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1668663 T3233 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1668663 T3233 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1668664 T3233 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1668664 T3233 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1668665 T3233 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1668665 T3233 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1668665 T3233 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1668666 T3233 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1668675 T3233 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1668679 T3233 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1668679 T3233 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1668680 T3233 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=69.9013671875, floorSegmentMB=0.7578125, forceMergeDeletesPctAllowed=3.0481504002723367, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1668681 T3233 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 1668682 T3233 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1668682 T3233 oass.SolrIndexSearcher.<init> Opening Searcher@7185d704 main
[junit4] 2> 1668685 T3234 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7185d704 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1668686 T3233 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1668687 T3204 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4] 2> 1668688 T3204 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1668687 T3237 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:28670 collection:control_collection shard:shard1
[junit4] 2> 1668689 T3237 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 1668691 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1668702 T3240 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78bc865c name:ZooKeeperConnection Watcher:127.0.0.1:42233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1668702 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1668704 T3204 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1668719 T3237 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1668719 T3204 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1668722 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1668722 T3237 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1668722 T3237 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1216 name=collection1 org.apache.solr.core.SolrCore@4475f8a0 url=http://127.0.0.1:28670/collection1 node=127.0.0.1:28670_ C1216_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:28670, core=collection1, node_name=127.0.0.1:28670_}
[junit4] 2> 1668723 T3237 C1216 P28670 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:28670/collection1/
[junit4] 2> 1668723 T3237 C1216 P28670 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1668723 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1668723 T3237 C1216 P28670 oasc.SyncStrategy.syncToMe http://127.0.0.1:28670/collection1/ has no replicas
[junit4] 2> 1668724 T3237 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:28670/collection1/ shard1
[junit4] 2> 1668724 T3237 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 1668728 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1668739 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1668744 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1668759 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1668764 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1668868 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1668868 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1668912 T3237 oasc.ZkController.register We are http://127.0.0.1:28670/collection1/ and leader is http://127.0.0.1:28670/collection1/
[junit4] 2> 1668912 T3237 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:28670
[junit4] 2> 1668912 T3237 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1668912 T3237 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1668912 T3237 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1668914 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1668914 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1668914 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1668915 T3237 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1668916 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1668917 T3231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:28670",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:28670_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1668920 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1669004 T3204 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1669006 T3204 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:37265
[junit4] 2> 1669007 T3204 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1669008 T3204 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1669008 T3204 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471
[junit4] 2> 1669009 T3204 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471/'
[junit4] 2> 1669024 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1669024 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1669050 T3204 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471/solr.xml
[junit4] 2> 1669130 T3204 oasc.CoreContainer.<init> New CoreContainer 822232844
[junit4] 2> 1669131 T3204 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471/]
[junit4] 2> 1669133 T3204 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1669133 T3204 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1669134 T3204 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1669134 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1669134 T3204 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1669135 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1669135 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1669136 T3204 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1669136 T3204 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1669154 T3204 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1669155 T3204 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1669155 T3204 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1669156 T3204 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42233/solr
[junit4] 2> 1669156 T3204 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1669158 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1669185 T3252 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51b7e98d name:ZooKeeperConnection Watcher:127.0.0.1:42233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1669186 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1669190 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1669200 T3255 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a26c38c name:ZooKeeperConnection Watcher:127.0.0.1:42233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1669200 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1669206 T3204 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1670209 T3204 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37265_
[junit4] 2> 1670219 T3204 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37265_
[junit4] 2> 1670224 T3240 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1670224 T3255 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1670224 T3230 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1670239 T3256 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1670240 T3256 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1670241 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1670241 T3256 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1670242 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1670242 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1670243 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1670244 T3231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:37265",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:37265_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1670244 T3231 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 1670244 T3231 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1670248 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1670387 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1670388 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1670387 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1671243 T3256 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1671243 T3256 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471/collection1
[junit4] 2> 1671243 T3256 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1671244 T3256 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1671245 T3256 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1671246 T3256 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471/collection1/'
[junit4] 2> 1671248 T3256 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471/collection1/lib/classes/' to classloader
[junit4] 2> 1671249 T3256 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471/collection1/lib/README' to classloader
[junit4] 2> 1671312 T3256 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 1671376 T3256 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1671477 T3256 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1671499 T3256 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1672077 T3256 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1672081 T3256 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1672084 T3256 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1672089 T3256 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1672119 T3256 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1672119 T3256 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty1/
[junit4] 2> 1672120 T3256 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4e14093f
[junit4] 2> 1672121 T3256 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty1
[junit4] 2> 1672122 T3256 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty1/index/
[junit4] 2> 1672123 T3256 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 1672124 T3256 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty1/index
[junit4] 2> 1672124 T3256 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=316730711, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 1672165 T3256 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1672166 T3256 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1672170 T3256 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1672171 T3256 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1672171 T3256 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1672171 T3256 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1672172 T3256 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1672172 T3256 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1672173 T3256 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1672173 T3256 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1672173 T3256 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1672174 T3256 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1672175 T3256 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1672175 T3256 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1672175 T3256 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1672176 T3256 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1672176 T3256 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1672177 T3256 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1672187 T3256 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1672191 T3256 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1672192 T3256 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1672193 T3256 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=69.9013671875, floorSegmentMB=0.7578125, forceMergeDeletesPctAllowed=3.0481504002723367, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1672194 T3256 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1672195 T3256 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1672195 T3256 oass.SolrIndexSearcher.<init> Opening Searcher@7843566e main
[junit4] 2> 1672199 T3257 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7843566e main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1672201 T3256 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1672202 T3204 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4] 2> 1672203 T3204 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1672202 T3260 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37265 collection:collection1 shard:shard1
[junit4] 2> 1672204 T3260 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1672245 T3260 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1672256 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1672257 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1672257 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1672257 T3260 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1672257 T3260 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1217 name=collection1 org.apache.solr.core.SolrCore@6327c579 url=http://127.0.0.1:37265/collection1 node=127.0.0.1:37265_ C1217_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:37265, core=collection1, node_name=127.0.0.1:37265_}
[junit4] 2> 1672258 T3260 C1217 P37265 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:37265/collection1/
[junit4] 2> 1672258 T3260 C1217 P37265 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1672258 T3260 C1217 P37265 oasc.SyncStrategy.syncToMe http://127.0.0.1:37265/collection1/ has no replicas
[junit4] 2> 1672258 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1672259 T3260 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:37265/collection1/ shard1
[junit4] 2> 1672259 T3260 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1672297 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1672340 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1672340 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1672340 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1672381 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1672385 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1672491 T3204 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1672493 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1672494 T3204 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:18099
[junit4] 2> 1672493 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1672495 T3204 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1672493 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1672496 T3204 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1672496 T3204 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956
[junit4] 2> 1672497 T3204 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956/'
[junit4] 2> 1672533 T3260 oasc.ZkController.register We are http://127.0.0.1:37265/collection1/ and leader is http://127.0.0.1:37265/collection1/
[junit4] 2> 1672533 T3260 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37265
[junit4] 2> 1672533 T3260 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1672533 T3260 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1672533 T3260 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1672535 T3204 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956/solr.xml
[junit4] 2> 1672537 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1672537 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1672538 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1672538 T3260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1672539 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1672540 T3231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:37265",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:37265_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1672543 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1672613 T3204 oasc.CoreContainer.<init> New CoreContainer 1441207837
[junit4] 2> 1672614 T3204 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956/]
[junit4] 2> 1672616 T3204 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1672616 T3204 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1672617 T3204 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1672617 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1672618 T3204 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1672618 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1672619 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1672619 T3204 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1672619 T3204 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1672637 T3204 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1672638 T3204 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1672639 T3204 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1672639 T3204 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42233/solr
[junit4] 2> 1672640 T3204 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1672641 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1672648 T3272 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48da33b6 name:ZooKeeperConnection Watcher:127.0.0.1:42233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1672649 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1672680 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1672680 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1672680 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1672709 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1672725 T3275 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78dd552e name:ZooKeeperConnection Watcher:127.0.0.1:42233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1672725 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1672730 T3204 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1673734 T3204 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:18099_
[junit4] 2> 1673763 T3204 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:18099_
[junit4] 2> 1673766 T3240 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1673767 T3230 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1673767 T3275 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1673767 T3255 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1673783 T3276 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1673783 T3276 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1673785 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1673785 T3276 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1673785 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1673785 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1673786 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1673787 T3231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:18099",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:18099_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1673787 T3231 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1673788 T3231 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1673791 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1673894 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1673894 T3275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1673894 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1673894 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1674786 T3276 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1674786 T3276 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956/collection1
[junit4] 2> 1674786 T3276 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1674787 T3276 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1674788 T3276 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1674789 T3276 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956/collection1/'
[junit4] 2> 1674790 T3276 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956/collection1/lib/README' to classloader
[junit4] 2> 1674791 T3276 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956/collection1/lib/classes/' to classloader
[junit4] 2> 1674856 T3276 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 1674921 T3276 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1675022 T3276 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1675044 T3276 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1675678 T3276 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1675683 T3276 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1675686 T3276 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1675692 T3276 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1675717 T3276 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1675718 T3276 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty2/
[junit4] 2> 1675718 T3276 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4e14093f
[junit4] 2> 1675720 T3276 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty2
[junit4] 2> 1675721 T3276 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty2/index/
[junit4] 2> 1675722 T3276 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 1675724 T3276 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty2/index
[junit4] 2> 1675725 T3276 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=316730711, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 1675729 T3276 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1675730 T3276 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1675734 T3276 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1675734 T3276 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1675735 T3276 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1675735 T3276 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1675735 T3276 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1675736 T3276 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1675736 T3276 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1675737 T3276 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1675737 T3276 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1675738 T3276 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1675738 T3276 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1675738 T3276 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1675739 T3276 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1675739 T3276 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1675740 T3276 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1675741 T3276 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1675754 T3276 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1675758 T3276 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1675759 T3276 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1675760 T3276 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=69.9013671875, floorSegmentMB=0.7578125, forceMergeDeletesPctAllowed=3.0481504002723367, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1675762 T3276 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1675762 T3276 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1675762 T3276 oass.SolrIndexSearcher.<init> Opening Searcher@3aebe998 main
[junit4] 2> 1675767 T3277 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3aebe998 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1675769 T3276 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1675770 T3204 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4] 2> 1675770 T3204 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1675770 T3280 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:18099 collection:collection1 shard:shard2
[junit4] 2> 1675772 T3280 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1675788 T3280 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1675796 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1675796 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1675796 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1675796 T3280 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1675797 T3280 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1218 name=collection1 org.apache.solr.core.SolrCore@1b62e6e5 url=http://127.0.0.1:18099/collection1 node=127.0.0.1:18099_ C1218_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:18099, core=collection1, node_name=127.0.0.1:18099_}
[junit4] 2> 1675797 T3280 C1218 P18099 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:18099/collection1/
[junit4] 2> 1675797 T3280 C1218 P18099 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1675797 T3280 C1218 P18099 oasc.SyncStrategy.syncToMe http://127.0.0.1:18099/collection1/ has no replicas
[junit4] 2> 1675798 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1675798 T3280 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:18099/collection1/ shard2
[junit4] 2> 1675798 T3280 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1675802 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1675811 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1675811 T3275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1675811 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1675811 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1675823 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1675832 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1675935 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1675935 T3275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1675935 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1675935 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1675974 T3280 oasc.ZkController.register We are http://127.0.0.1:18099/collection1/ and leader is http://127.0.0.1:18099/collection1/
[junit4] 2> 1675974 T3280 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:18099
[junit4] 2> 1675974 T3280 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1675974 T3280 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1675975 T3280 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1675976 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1675977 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1675977 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1675977 T3280 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1675979 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1675979 T3231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:18099",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:18099_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 1675983 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1676088 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1676089 T3275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1676089 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1676088 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1676120 T3204 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1676123 T3204 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:47019
[junit4] 2> 1676123 T3204 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1676124 T3204 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1676124 T3204 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1388276358524
[junit4] 2> 1676125 T3204 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1388276358524/'
[junit4] 2> 1676161 T3204 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1388276358524/solr.xml
[junit4] 2> 1676234 T3204 oasc.CoreContainer.<init> New CoreContainer 498415665
[junit4] 2> 1676234 T3204 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1388276358524/]
[junit4] 2> 1676236 T3204 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1676237 T3204 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1676237 T3204 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1676237 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1676238 T3204 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1676238 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1676239 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1676239 T3204 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1676240 T3204 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1676255 T3204 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1676255 T3204 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1676256 T3204 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1676256 T3204 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42233/solr
[junit4] 2> 1676257 T3204 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1676258 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1676260 T3292 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13335fcf name:ZooKeeperConnection Watcher:127.0.0.1:42233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1676261 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1676265 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1676272 T3295 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21a6c2a6 name:ZooKeeperConnection Watcher:127.0.0.1:42233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1676272 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1676277 T3204 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1677281 T3204 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47019_
[junit4] 2> 1677284 T3204 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47019_
[junit4] 2> 1677288 T3240 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1677288 T3230 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1677288 T3275 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1677288 T3255 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1677289 T3295 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1677298 T3296 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1677298 T3296 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1677300 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1677300 T3296 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1677300 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1677301 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1677301 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1677302 T3231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:47019",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:47019_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1677303 T3231 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1677303 T3231 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1677306 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1677408 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1677409 T3275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1677409 T3295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1677409 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1677408 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1678301 T3296 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1678301 T3296 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1388276358524/collection1
[junit4] 2> 1678301 T3296 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1678302 T3296 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1678302 T3296 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1678303 T3296 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1388276358524/collection1/'
[junit4] 2> 1678305 T3296 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1388276358524/collection1/lib/classes/' to classloader
[junit4] 2> 1678305 T3296 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1388276358524/collection1/lib/README' to classloader
[junit4] 2> 1678349 T3296 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 1678390 T3296 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1678492 T3296 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1678514 T3296 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1679103 T3296 oass.IndexSchema.readSchema
[...truncated too long message...]
lectionContext.runLeaderProcess(ElectionContext.java:258)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4] 2>
[junit4] 2> 1820340 T3382 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=27,cumulative_deletesById=13,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=1}
[junit4] 2> 1820341 T3382 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 1820341 T3382 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 1820341 T3382 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 1820342 T3382 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 1820343 T3382 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 1820343 T3382 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty7/index [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty7/index;done=false>>]
[junit4] 2> 1820343 T3382 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty7/index
[junit4] 2> 1820344 T3382 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty7 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty7;done=false>>]
[junit4] 2> 1820344 T3382 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty7
[junit4] 2> 1820344 T3382 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1820344 T3382 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1820345 T3382 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1820345 T3382 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4] 2>
[junit4] 2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=fi, timezone=Etc/UCT
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=275901600,total=460980224
[junit4] 2> NOTE: All tests run in this JVM: [TestIndexingPerformance, EnumFieldTest, TestFoldingMultitermQuery, TestQuerySenderNoQuery, SolrCmdDistributorTest, TestReload, BasicZkTest, UniqFieldsUpdateProcessorFactoryTest, BasicDistributedZk2Test, TestAtomicUpdateErrorCases, TestDistributedMissingSort, SolrIndexConfigTest, AddBlockUpdateTest, TestStressRecovery, CurrencyFieldOpenExchangeTest, TestInfoStreamLogging, TestDefaultSearchFieldResource, TestDistributedGrouping, DistributedQueryComponentCustomSortTest, TestSolrQueryParserDefaultOperatorResource, DirectUpdateHandlerTest, TestRangeQuery, TestMultiCoreConfBootstrap, TestCSVLoader, TestQuerySenderListener, ZkSolrClientTest, CachingDirectoryFactoryTest, SortByFunctionTest, TestComponentsName, SuggesterWFSTTest, ConnectionManagerTest, TestQueryTypes, TestFiltering, ConvertedLegacyTest, SuggestComponentTest, TestFieldTypeResource, FieldAnalysisRequestHandlerTest, OpenCloseCoreStressTest, MigrateRouteKeyTest, XsltUpdateRequestHandlerTest, TestSolrQueryParserResource, TestRecovery, TestFuzzyAnalyzedSuggestions, TestSolr4Spatial, BlockDirectoryTest, TestFastOutputStream, LeaderElectionIntegrationTest, RAMDirectoryFactoryTest, ZkNodePropsTest, NoCacheHeaderTest, TermVectorComponentDistributedTest, DisMaxRequestHandlerTest, TestSchemaResource, MultiTermTest, TestSolrXmlPersistence, RequestHandlersTest, HdfsChaosMonkeySafeLeaderTest, SolrCoreCheckLockOnStartupTest, DocumentAnalysisRequestHandlerTest, TestStressLucene, PrimUtilsTest, PathHierarchyTokenizerFactoryTest, RequiredFieldsTest, TestSolrJ, SpellCheckComponentTest, SolrPluginUtilsTest, TestHashPartitioner, SimplePostToolTest, TestIBSimilarityFactory, TestMaxScoreQueryParser, AliasIntegrationTest, TestDistribDocBasedVersion, PreAnalyzedUpdateProcessorTest, NumericFieldsTest, IndexSchemaRuntimeFieldTest, SpellingQueryConverterTest, TestLazyCores, TestSolrCoreProperties, TestSchemaVersionResource, TestCollapseQParserPlugin, TestStressReorder, TestFieldResource, OpenExchangeRatesOrgProviderTest, TestBinaryResponseWriter, MinimalSchemaTest, ClusterStateUpdateTest, TestRandomFaceting, TestFaceting, ZkControllerTest, TestRealTimeGet, HardAutoCommitTest, DistributedTermsComponentTest, StatsComponentTest, TestGroupingSearch, HighlighterTest, DistributedQueryElevationComponentTest, TestCoreDiscovery, SignatureUpdateProcessorFactoryTest, SpellCheckCollatorTest, SolrRequestParserTest, SuggesterTest, WordBreakSolrSpellCheckerTest, TestUpdate, TestWordDelimiterFilterFactory, DefaultValueUpdateProcessorTest, DistanceFunctionTest, CacheHeaderTest, TestSurroundQueryParser, DirectSolrSpellCheckerTest, FileBasedSpellCheckerTest, TestValueSourceCache, FastVectorHighlighterTest, IndexSchemaTest, TestCollationField, JSONWriterTest, MBeansHandlerTest, JsonLoaderTest, TestPartialUpdateDeduplication, PingRequestHandlerTest, CoreContainerCoreInitFailuresTest, HighlighterConfigTest, AlternateDirectoryTest, TestSolrIndexConfig, ResponseLogComponentTest, SampleTest, TestConfig, TestPostingsSolrHighlighter, NotRequiredUniqueKeyTest, TestPhraseSuggestions, TestXIncludeConfig, TestDFRSimilarityFactory, TestNumberUtils, TestLMDirichletSimilarityFactory, TestDefaultSimilarityFactory, LegacyHTMLStripCharFilterTest, ScriptEngineTest, TestFastLRUCache, ChaosMonkeyNothingIsSafeTest]
[junit4] Completed on J1 in 155.36s, 1 test, 1 failure <<< FAILURES!
[...truncated 130 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:459: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:439: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/common-build.xml:491: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1307: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:940: There were test failures: 348 suites, 1519 tests, 1 failure, 35 ignored (6 assumptions)
Total time: 60 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 2243 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/2243/
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard2 is not consistent. Got 134 from http://127.0.0.1:56787/collection1lastClient and got 135 from http://127.0.0.1:56808/collection1
Stack Trace:
java.lang.AssertionError: shard2 is not consistent. Got 134 from http://127.0.0.1:56787/collection1lastClient and got 135 from http://127.0.0.1:56808/collection1
at __randomizedtesting.SeedInfo.seed([A921DE79F7B1E6D4:28C7506180EE86E8]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1152)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:149)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:849)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:679)
Build Log:
[...truncated 9783 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> 792425 T4157 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 792431 T4157 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeySafeLeaderTest-1388321240224
[junit4] 2> 792432 T4157 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 792433 T4158 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 792534 T4157 oasc.ZkTestServer.run start zk server on port:56723
[junit4] 2> 792535 T4157 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 792564 T4165 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bddce24 name:ZooKeeperConnection Watcher:127.0.0.1:56723 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 792564 T4157 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 792565 T4157 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 792576 T4157 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 792578 T4168 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@73ab5b1d name:ZooKeeperConnection Watcher:127.0.0.1:56723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 792578 T4157 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 792578 T4157 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 792588 T4157 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 792591 T4157 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 792593 T4157 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 792602 T4157 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 792603 T4157 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 792616 T4157 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 792616 T4157 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 792720 T4157 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 792720 T4157 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 792724 T4157 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 792724 T4157 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 792732 T4157 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 792733 T4157 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 792736 T4157 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 792736 T4157 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 792739 T4157 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 792740 T4157 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 792743 T4157 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 792743 T4157 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 792751 T4157 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 792752 T4157 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 792754 T4157 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 792755 T4157 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 792758 T4157 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 792759 T4157 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 793051 T4157 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 793056 T4157 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:56726
[junit4] 2> 793057 T4157 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 793057 T4157 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 793058 T4157 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388321240555
[junit4] 2> 793058 T4157 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388321240555/'
[junit4] 2> 793101 T4157 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388321240555/solr.xml
[junit4] 2> 793185 T4157 oasc.CoreContainer.<init> New CoreContainer 805920918
[junit4] 2> 793186 T4157 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388321240555/]
[junit4] 2> 793188 T4157 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 793188 T4157 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 793189 T4157 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 793189 T4157 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 793190 T4157 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 793190 T4157 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 793191 T4157 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 793191 T4157 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 793192 T4157 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 793210 T4157 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 793211 T4157 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 793211 T4157 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 793212 T4157 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56723/solr
[junit4] 2> 793212 T4157 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 793213 T4157 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 793216 T4180 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c6b7b14 name:ZooKeeperConnection Watcher:127.0.0.1:56723 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 793216 T4157 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 793219 T4157 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 793227 T4183 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c3a7aea name:ZooKeeperConnection Watcher:127.0.0.1:56723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 793227 T4157 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 793229 T4157 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 793232 T4157 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 793242 T4157 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 793244 T4157 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56726_
[junit4] 2> 793246 T4157 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56726_
[junit4] 2> 793254 T4157 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 793270 T4157 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 793272 T4157 oasc.Overseer.start Overseer (id=90985020799778819-127.0.0.1:56726_-n_0000000000) starting
[junit4] 2> 793284 T4157 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 793293 T4185 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 793293 T4157 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 793301 T4157 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 793303 T4157 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 793305 T4184 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 793311 T4186 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 793311 T4186 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 793312 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 793312 T4186 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 793313 T4184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 793314 T4184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:56726",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:56726_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 793314 T4184 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 793315 T4184 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 793317 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 793327 T4183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 794314 T4186 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 794314 T4186 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388321240555/collection1
[junit4] 2> 794314 T4186 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 794315 T4186 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 794315 T4186 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 794316 T4186 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388321240555/collection1/'
[junit4] 2> 794318 T4186 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388321240555/collection1/lib/classes/' to classloader
[junit4] 2> 794318 T4186 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388321240555/collection1/lib/README' to classloader
[junit4] 2> 794388 T4186 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 794452 T4186 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 794553 T4186 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 794563 T4186 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 795175 T4186 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 795179 T4186 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 795182 T4186 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 795187 T4186 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 795209 T4186 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 795209 T4186 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388321240555/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/control/data/
[junit4] 2> 795209 T4186 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@74b2e5d0
[junit4] 2> 795211 T4186 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/control/data
[junit4] 2> 795212 T4186 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/control/data/index/
[junit4] 2> 795213 T4186 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 795215 T4186 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/control/data/index
[junit4] 2> 795215 T4186 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=35, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4989817952394622]
[junit4] 2> 795256 T4186 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 795256 T4186 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 795259 T4186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 795260 T4186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 795260 T4186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 795260 T4186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 795261 T4186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 795261 T4186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 795262 T4186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 795262 T4186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 795262 T4186 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 795263 T4186 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 795263 T4186 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 795264 T4186 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 795264 T4186 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 795265 T4186 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 795265 T4186 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 795266 T4186 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 795277 T4186 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 795281 T4186 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 795282 T4186 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 795283 T4186 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1247527787, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 795285 T4186 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 795285 T4186 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 795286 T4186 oass.SolrIndexSearcher.<init> Opening Searcher@1aafe09e main
[junit4] 2> 795289 T4187 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1aafe09e main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 795290 T4186 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 795291 T4157 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4] 2> 795291 T4157 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 795291 T4190 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56726 collection:control_collection shard:shard1
[junit4] 2> 795293 T4190 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 795294 T4157 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 795296 T4193 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b6f333d name:ZooKeeperConnection Watcher:127.0.0.1:56723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 795297 T4157 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 795306 T4157 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 795319 T4190 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 795319 T4157 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 795327 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 795327 T4190 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 795327 T4190 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C838 name=collection1 org.apache.solr.core.SolrCore@4e1f649c url=http://127.0.0.1:56726/collection1 node=127.0.0.1:56726_ C838_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:56726, core=collection1, node_name=127.0.0.1:56726_}
[junit4] 2> 795328 T4190 C838 P56726 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56726/collection1/
[junit4] 2> 795328 T4184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 795328 T4190 C838 P56726 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 795329 T4190 C838 P56726 oasc.SyncStrategy.syncToMe http://127.0.0.1:56726/collection1/ has no replicas
[junit4] 2> 795329 T4190 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56726/collection1/ shard1
[junit4] 2> 795329 T4190 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 795332 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 795340 T4183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 795340 T4193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 795352 T4184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 795365 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 795468 T4193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 795468 T4183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 795505 T4190 oasc.ZkController.register We are http://127.0.0.1:56726/collection1/ and leader is http://127.0.0.1:56726/collection1/
[junit4] 2> 795505 T4190 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56726
[junit4] 2> 795505 T4190 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 795505 T4190 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 795505 T4190 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 795507 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 795507 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 795507 T4190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 795507 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 795508 T4184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 795509 T4184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:56726",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:56726_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 795512 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 795600 T4157 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 795603 T4157 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:56736
[junit4] 2> 795603 T4157 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 795604 T4157 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 795604 T4157 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388321243111
[junit4] 2> 795605 T4157 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388321243111/'
[junit4] 2> 795614 T4183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 795614 T4193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 795643 T4157 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388321243111/solr.xml
[junit4] 2> 795731 T4157 oasc.CoreContainer.<init> New CoreContainer 172488615
[junit4] 2> 795732 T4157 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388321243111/]
[junit4] 2> 795734 T4157 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 795734 T4157 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 795735 T4157 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 795735 T4157 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 795736 T4157 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 795736 T4157 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 795737 T4157 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 795737 T4157 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 795738 T4157 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 795754 T4157 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 795755 T4157 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 795755 T4157 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 795756 T4157 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56723/solr
[junit4] 2> 795756 T4157 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 795758 T4157 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 795771 T4205 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72d02e39 name:ZooKeeperConnection Watcher:127.0.0.1:56723 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 795772 T4157 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 795787 T4157 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 795826 T4208 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19119096 name:ZooKeeperConnection Watcher:127.0.0.1:56723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 795826 T4157 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 795830 T4157 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 796833 T4157 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56736_
[junit4] 2> 796835 T4157 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56736_
[junit4] 2> 796848 T4193 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 796848 T4183 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 796848 T4208 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 796862 T4209 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 796862 T4209 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 796864 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 796864 T4209 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 796864 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 796864 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 796865 T4184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 796866 T4184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:56736",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:56736_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 796866 T4184 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 796867 T4184 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 796875 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 796978 T4183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 796978 T4208 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 796978 T4193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 797865 T4209 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 797865 T4209 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388321243111/collection1
[junit4] 2> 797865 T4209 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 797866 T4209 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 797866 T4209 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 797867 T4209 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388321243111/collection1/'
[junit4] 2> 797868 T4209 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388321243111/collection1/lib/classes/' to classloader
[junit4] 2> 797869 T4209 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388321243111/collection1/lib/README' to classloader
[junit4] 2> 797910 T4209 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 797952 T4209 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 798053 T4209 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 798064 T4209 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 798686 T4209 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 798691 T4209 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 798693 T4209 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 798698 T4209 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 798719 T4209 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 798719 T4209 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388321243111/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty1/
[junit4] 2> 798719 T4209 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@74b2e5d0
[junit4] 2> 798721 T4209 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty1
[junit4] 2> 798722 T4209 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty1/index/
[junit4] 2> 798722 T4209 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 798723 T4209 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty1/index
[junit4] 2> 798724 T4209 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=35, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4989817952394622]
[junit4] 2> 798728 T4209 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 798729 T4209 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 798732 T4209 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 798733 T4209 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 798733 T4209 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 798734 T4209 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 798734 T4209 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 798734 T4209 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 798735 T4209 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 798735 T4209 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 798735 T4209 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 798736 T4209 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 798737 T4209 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 798737 T4209 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 798737 T4209 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 798738 T4209 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 798738 T4209 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 798739 T4209 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 798749 T4209 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 798753 T4209 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 798753 T4209 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 798754 T4209 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1247527787, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 798756 T4209 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 798756 T4209 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 798757 T4209 oass.SolrIndexSearcher.<init> Opening Searcher@14891426 main
[junit4] 2> 798761 T4210 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@14891426 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 798763 T4209 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 798764 T4157 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4] 2> 798765 T4157 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 798764 T4213 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56736 collection:collection1 shard:shard1
[junit4] 2> 798766 T4213 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 798782 T4213 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 798792 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 798792 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 798793 T4213 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 798793 T4213 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 798793 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C839 name=collection1 org.apache.solr.core.SolrCore@3b5f71d2 url=http://127.0.0.1:56736/collection1 node=127.0.0.1:56736_ C839_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:56736, core=collection1, node_name=127.0.0.1:56736_}
[junit4] 2> 798793 T4213 C839 P56736 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56736/collection1/
[junit4] 2> 798794 T4184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 798794 T4213 C839 P56736 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 798794 T4213 C839 P56736 oasc.SyncStrategy.syncToMe http://127.0.0.1:56736/collection1/ has no replicas
[junit4] 2> 798794 T4213 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56736/collection1/ shard1
[junit4] 2> 798795 T4213 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 798797 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 798807 T4183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 798807 T4193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 798807 T4208 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 798819 T4184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 798829 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 798931 T4183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 798931 T4193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 798931 T4208 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 798972 T4213 oasc.ZkController.register We are http://127.0.0.1:56736/collection1/ and leader is http://127.0.0.1:56736/collection1/
[junit4] 2> 798972 T4213 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56736
[junit4] 2> 798972 T4213 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 798972 T4213 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 798973 T4213 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 798974 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 798974 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 798974 T4213 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 798974 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 798975 T4184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 798976 T4184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:56736",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:56736_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 798985 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 799038 T4157 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 799041 T4157 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:56749
[junit4] 2> 799042 T4157 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 799042 T4157 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 799043 T4157 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388321246558
[junit4] 2> 799043 T4157 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388321246558/'
[junit4] 2> 799081 T4157 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388321246558/solr.xml
[junit4] 2> 799088 T4183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 799088 T4208 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 799088 T4193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 799160 T4157 oasc.CoreContainer.<init> New CoreContainer 612150909
[junit4] 2> 799161 T4157 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388321246558/]
[junit4] 2> 799162 T4157 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 799163 T4157 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 799163 T4157 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 799164 T4157 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 799164 T4157 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 799165 T4157 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 799165 T4157 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 799166 T4157 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 799166 T4157 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 799183 T4157 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 799184 T4157 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 799184 T4157 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 799185 T4157 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56723/solr
[junit4] 2> 799186 T4157 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 799187 T4157 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 799189 T4225 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a333cbd name:ZooKeeperConnection Watcher:127.0.0.1:56723 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 799189 T4157 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 799192 T4157 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 799200 T4228 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b5dfdf1 name:ZooKeeperConnection Watcher:127.0.0.1:56723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 799200 T4157 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 799204 T4157 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 800207 T4157 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56749_
[junit4] 2> 800209 T4157 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56749_
[junit4] 2> 800212 T4193 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 800212 T4208 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 800212 T4228 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 800212 T4183 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 800222 T4229 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 800222 T4229 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 800224 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 800224 T4229 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 800224 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 800225 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 800226 T4184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 800226 T4184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:56749",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:56749_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 800227 T4184 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 800227 T4184 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 800230 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 800332 T4208 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 800332 T4228 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 800332 T4193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 800332 T4183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 801225 T4229 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 801225 T4229 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388321246558/collection1
[junit4] 2> 801225 T4229 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 801226 T4229 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 801226 T4229 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 801227 T4229 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388321246558/collection1/'
[junit4] 2> 801228 T4229 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388321246558/collection1/lib/README' to classloader
[junit4] 2> 801229 T4229 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388321246558/collection1/lib/classes/' to classloader
[junit4] 2> 801271 T4229 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 801334 T4229 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 801436 T4229 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 801443 T4229 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 801985 T4229 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 801990 T4229 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 801992 T4229 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 801997 T4229 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 802018 T4229 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 802019 T4229 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388321246558/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty2/
[junit4] 2> 802019 T4229 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@74b2e5d0
[junit4] 2> 802020 T4229 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty2
[junit4] 2> 802021 T4229 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty2/index/
[junit4] 2> 802022 T4229 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 802023 T4229 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty2/index
[junit4] 2> 802023 T4229 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=35, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4989817952394622]
[junit4] 2> 802027 T4229 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 802027 T4229 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 802031 T4229 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 802031 T4229 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 802032 T4229 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 802032 T4229 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 802032 T4229 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 802033 T4229 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 802033 T4229 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 802034 T4229 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 802034 T4229 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 802034 T4229 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 802035 T4229 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 802035 T4229 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 802036 T4229 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 802036 T4229 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 802037 T4229 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 802037 T4229 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 802046 T4229 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 802050 T4229 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 802050 T4229 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 802051 T4229 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1247527787, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 802053 T4229 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 802053 T4229 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 802053 T4229 oass.SolrIndexSearcher.<init> Opening Searcher@140e3b4c main
[junit4] 2> 802057 T4230 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@140e3b4c main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 802059 T4229 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 802060 T4157 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4] 2> 802061 T4157 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 802060 T4233 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56749 collection:collection1 shard:shard2
[junit4] 2> 802062 T4233 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 802075 T4233 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 802084 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 802084 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 802084 T4233 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 802085 T4233 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 802084 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C840 name=collection1 org.apache.solr.core.SolrCore@13fd80d4 url=http://127.0.0.1:56749/collection1 node=127.0.0.1:56749_ C840_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:56749, core=collection1, node_name=127.0.0.1:56749_}
[junit4] 2> 802085 T4233 C840 P56749 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56749/collection1/
[junit4] 2> 802085 T4184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 802085 T4233 C840 P56749 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 802086 T4233 C840 P56749 oasc.SyncStrategy.syncToMe http://127.0.0.1:56749/collection1/ has no replicas
[junit4] 2> 802086 T4233 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56749/collection1/ shard2
[junit4] 2> 802086 T4233 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 802088 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 802089 T4183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 802090 T4228 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 802089 T4208 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 802089 T4193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 802093 T4184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 802096 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 802198 T4183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 802198 T4193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 802198 T4208 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 802198 T4228 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 802245 T4233 oasc.ZkController.register We are http://127.0.0.1:56749/collection1/ and leader is http://127.0.0.1:56749/collection1/
[junit4] 2> 802245 T4233 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56749
[junit4] 2> 802245 T4233 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 802245 T4233 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 802245 T4233 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 802247 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 802247 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 802247 T4233 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 802247 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 802248 T4184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 802249 T4184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:56749",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:56749_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 802257 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 802340 T4157 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 802342 T4157 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:56756
[junit4] 2> 802343 T4157 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 802344 T4157 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 802344 T4157 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388321249854
[junit4] 2> 802345 T4157 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388321249854/'
[junit4] 2> 802360 T4183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 802360 T4228 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 802360 T4193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 802360 T4208 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 802383 T4157 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388321249854/solr.xml
[junit4] 2> 802461 T4157 oasc.CoreContainer.<init> New CoreContainer 480940571
[junit4] 2> 802462 T4157 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388321249854/]
[junit4] 2> 802464 T4157 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 802464 T4157 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 802465 T4157 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 802465 T4157 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 802465 T4157 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 802466 T4157 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 802466 T4157 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 802467 T4157 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 802467 T4157 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 802485 T4157 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 802485 T4157 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 802486 T4157 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 802486 T4157 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56723/solr
[junit4] 2> 802487 T4157 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 802488 T4157 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 802490 T4245 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7804f60d name:ZooKeeperConnection Watcher:127.0.0.1:56723 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 802490 T4157 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 802493 T4157 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 802500 T4248 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10572374 name:ZooKeeperConnection Watcher:127.0.0.1:56723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 802500 T4157 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 802505 T4157 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 803508 T4157 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56756_
[junit4] 2> 803519 T4157 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56756_
[junit4] 2> 803573 T4183 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 803573 T4248 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 803573 T4208 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 803573 T4228 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 803573 T4193 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 803581 T4249 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 803582 T4249 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 803583 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 803583 T4249 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 803583 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 803584 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 803584 T4184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 803585 T4184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:56756",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:56756_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 803585 T4184 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 803585 T4184 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 803594 T4183 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 803697 T4248 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 803697 T4228 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 803697 T4193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 803697 T4183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 803697 T4208 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 804584 T4249 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 804585 T4249 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388321249854/collection1
[junit4] 2> 804585 T4249 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 804586 T4249 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 804586 T4249 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 804587 T4249 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388321249854/collection1/'
[junit4] 2> 804589 T4249 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388321249854/collection1/lib/classes/' to classloader
[junit4] 2> 804590 T4249 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388321249854/collection1/lib/README' to classloader
[junit4] 2> 804655 T4249 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 804720 T4249 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 804821 T4249 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 804831 T4249 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 805423 T4249 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 805427 T4249 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 805430 T4249 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 805435 T4249 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 805456 T4249 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 805456 T4249 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388321249854/collec
[...truncated too long message...]
che.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty11 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty11;done=false>>]
[junit4] 2> 933912 T4581 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty11
[junit4] 2> 933912 T4581 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 933913 T4581 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4] 2>
[junit4] 2> 933967 T4636 C1027 P35864 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
[junit4] 2> 933967 T4636 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
[junit4] 2> 933968 T4636 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@5e1ba12d
[junit4] 2> 933971 T4636 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=3,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=106,cumulative_deletesById=28,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=3}
[junit4] 2> 933972 T4636 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 933972 T4636 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 933972 T4636 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 933973 T4636 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 933974 T4636 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 933974 T4636 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty12/index.20131229054831185 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty12/index.20131229054831185;done=false>>]
[junit4] 2> 933974 T4636 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty12/index.20131229054831185
[junit4] 2> 933975 T4636 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty12 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty12;done=false>>]
[junit4] 2> 933975 T4636 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388321240223/jetty12
[junit4] 2> 933975 T4636 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 933975 T4636 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4] 2>
[junit4] 2> NOTE: test params are: codec=Asserting, sim=DefaultSimilarity, locale=es_CO, timezone=America/Denver
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=193808688,total=348585984
[junit4] 2> NOTE: All tests run in this JVM: [ResponseLogComponentTest, TestJoin, EchoParamsTest, TestNRTOpen, AddSchemaFieldsUpdateProcessorFactoryTest, CircularListTest, InfoHandlerTest, TestFieldTypeCollectionResource, SynonymTokenizerTest, BJQParserTest, FileBasedSpellCheckerTest, TestShardHandlerFactory, CopyFieldTest, UUIDFieldTest, DirectUpdateHandlerOptimizeTest, TermVectorComponentTest, BinaryUpdateRequestHandlerTest, IndexSchemaTest, ShardRoutingTest, ClusterStateTest, TriLevelCompositeIdRoutingTest, LukeRequestHandlerTest, TestStressVersions, ZkCLITest, TestTrie, DocValuesTest, HdfsBasicDistributedZk2Test, TestAddFieldRealTimeGet, TestCopyFieldCollectionResource, OverseerCollectionProcessorTest, TestLuceneMatchVersion, TestSurroundQueryParser, OverseerTest, TestSerializedLuceneMatchVersion, TestRecoveryHdfs, TestValueSourceCache, TestCustomSort, SolrTestCaseJ4Test, StatelessScriptUpdateProcessorFactoryTest, TestReplicationHandler, WordBreakSolrSpellCheckerTest, TestFileDictionaryLookup, TestRealTimeGet, PeerSyncTest, TestLFUCache, TestFunctionQuery, TestDocumentBuilder, TestCloudManagedSchema, QueryParsingTest, ChangedSchemaMergeTest, LoggingHandlerTest, TestSolrDeletionPolicy1, ParsingFieldUpdateProcessorsTest, DocumentBuilderTest, FastVectorHighlighterTest, URLClassifyProcessorTest, ShowFileRequestHandlerTest, TermsComponentTest, TestManagedSchema, DistributedSpellCheckComponentTest, ReturnFieldsTest, TestPseudoReturnFields, HdfsCollectionsAPIDistributedZkTest, TestPHPSerializedResponseWriter, TestSchemaNameResource, SolrCoreTest, LeaderElectionTest, MoreLikeThisHandlerTest, TestAnalyzeInfixSuggestions, TestLMJelinekMercerSimilarityFactory, TestReversedWildcardFilterFactory, TestRemoteStreaming, MBeansHandlerTest, ResourceLoaderTest, DOMUtilTest, TestQueryUtils, DeleteReplicaTest, CSVRequestHandlerTest, SearchHandlerTest, TestLRUCache, SampleTest, TestSolrXml, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest]
[junit4] Completed on J0 in 141.75s, 1 test, 1 failure <<< FAILURES!
[...truncated 610 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:459: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:439: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/common-build.xml:491: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1307: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:940: There were test failures: 348 suites, 1519 tests, 1 failure, 35 ignored (6 assumptions)
Total time: 69 minutes 59 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure