You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/05/07 07:10:02 UTC
[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.8.0) - Build # 1558
- Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1558/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC
2 tests failed.
REGRESSION: org.apache.solr.cloud.BasicDistributedZkTest.testDistribSearch
Error Message:
commitWithin did not work expected:<68> but was:<67>
Stack Trace:
java.lang.AssertionError: commitWithin did not work expected:<68> but was:<67>
at __randomizedtesting.SeedInfo.seed([8E71F0B3A8B73DF7:F977EABDFE85DCB]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.apache.solr.cloud.BasicDistributedZkTest.doTest(BasicDistributedZkTest.java:347)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
expected:<0> but was:<1>
Stack Trace:
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([8E71F0B3A8B73DF7:F977EABDFE85DCB]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:141)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11125 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/init-core-data-001
[junit4] 2> 993736 T4861 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 993737 T4861 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 993745 T4861 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 993747 T4861 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 993749 T4862 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 993850 T4861 oasc.ZkTestServer.run start zk server on port:50536
[junit4] 2> 993853 T4861 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 993863 T4868 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a3b2da5 name:ZooKeeperConnection Watcher:127.0.0.1:50536 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 993869 T4861 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 993870 T4861 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 993895 T4861 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 993902 T4870 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e158090 name:ZooKeeperConnection Watcher:127.0.0.1:50536/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 993902 T4861 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 993903 T4861 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 993910 T4861 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 993917 T4861 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 993924 T4861 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 993931 T4861 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 993932 T4861 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 993944 T4861 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 993945 T4861 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 993957 T4861 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 993958 T4861 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 993966 T4861 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 993966 T4861 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 993974 T4861 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 993974 T4861 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 993982 T4861 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 993982 T4861 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 993991 T4861 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 993992 T4861 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 994000 T4861 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 994001 T4861 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 994009 T4861 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 994009 T4861 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 994016 T4861 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 994017 T4861 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 994024 T4861 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 994024 T4861 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 994033 T4861 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 994037 T4872 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c634d08 name:ZooKeeperConnection Watcher:127.0.0.1:50536/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 994037 T4861 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 995750 T4861 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 995758 T4861 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 995765 T4861 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50540
[junit4] 2> 995767 T4861 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 995767 T4861 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 995768 T4861 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-002
[junit4] 2> 995768 T4861 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-002/'
[junit4] 2> 995811 T4861 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-002/solr.xml
[junit4] 2> 995842 T4861 oasc.CoreContainer.<init> New CoreContainer 174601210
[junit4] 2> 995842 T4861 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-002/]
[junit4] 2> 995843 T4861 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 995843 T4861 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 995844 T4861 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 995844 T4861 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 995845 T4861 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 995846 T4861 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 995846 T4861 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 995847 T4861 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 995847 T4861 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 995852 T4861 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 995852 T4861 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 995853 T4861 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 995853 T4861 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50536/solr
[junit4] 2> 995853 T4861 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 995857 T4861 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 995860 T4883 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29a34a11 name:ZooKeeperConnection Watcher:127.0.0.1:50536 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 995861 T4861 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 995866 T4861 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 995871 T4885 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9da2b42 name:ZooKeeperConnection Watcher:127.0.0.1:50536/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 995872 T4861 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 995876 T4861 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 995885 T4861 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 995892 T4861 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 995901 T4861 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 995910 T4861 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 995920 T4861 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 995925 T4861 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50540_
[junit4] 2> 995928 T4861 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50540_
[junit4] 2> 995936 T4861 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 995942 T4861 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 995953 T4861 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50540_
[junit4] 2> 995954 T4861 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 995959 T4861 oasc.Overseer.start Overseer (id=91713522805702660-127.0.0.1:50540_-n_0000000000) starting
[junit4] 2> 995967 T4861 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 996001 T4887 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 996002 T4861 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 996003 T4887 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:50540_
[junit4] 2> 996008 T4861 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 996015 T4861 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 996025 T4886 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 996031 T4888 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 996031 T4888 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 996034 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 996035 T4888 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 996039 T4886 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 996041 T4886 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:50540",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:50540_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 996041 T4886 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 996042 T4886 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 996049 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 996051 T4885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 997037 T4888 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 997037 T4888 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 997039 T4888 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 997039 T4888 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 997043 T4888 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 997044 T4888 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-002/collection1/'
[junit4] 2> 997045 T4888 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 997045 T4888 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 997045 T4888 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 997092 T4888 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 997105 T4888 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 997109 T4888 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 997114 T4888 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 997217 T4888 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 997218 T4888 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 997220 T4888 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 997225 T4888 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 997277 T4888 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 997277 T4888 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 997277 T4888 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/control/data/
[junit4] 2> 997277 T4888 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6994cbf
[junit4] 2> 997279 T4888 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/control/data
[junit4] 2> 997280 T4888 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/control/data/index/
[junit4] 2> 997281 T4888 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 997282 T4888 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/control/data/index
[junit4] 2> 997282 T4888 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=57.4130859375, floorSegmentMB=0.9599609375, forceMergeDeletesPctAllowed=29.834462607382324, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 997287 T4888 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 997287 T4888 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 997290 T4888 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 997291 T4888 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 997291 T4888 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 997291 T4888 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 997291 T4888 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 997292 T4888 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 997292 T4888 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 997292 T4888 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 997293 T4888 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 997293 T4888 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 997294 T4888 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 997294 T4888 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 997294 T4888 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 997295 T4888 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 997295 T4888 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 997296 T4888 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 997297 T4888 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 997303 T4888 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 997305 T4888 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 997305 T4888 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 997306 T4888 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=84.185546875, floorSegmentMB=1.3896484375, forceMergeDeletesPctAllowed=12.252324090316794, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4486034771152828
[junit4] 2> 997307 T4888 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 997308 T4888 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 997308 T4888 oass.SolrIndexSearcher.<init> Opening Searcher@7b279d8b[collection1] main
[junit4] 2> 997308 T4888 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 997310 T4888 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 997310 T4888 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 997311 T4888 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 997312 T4888 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 997312 T4888 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 997313 T4888 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 997313 T4888 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 997313 T4888 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 997314 T4888 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 997320 T4889 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7b279d8b[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 997321 T4888 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 997323 T4892 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50540 collection:control_collection shard:shard1
[junit4] 2> 997324 T4861 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 997325 T4861 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 997327 T4892 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 997334 T4861 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 997337 T4894 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b966fe5 name:ZooKeeperConnection Watcher:127.0.0.1:50536/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 997337 T4861 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 997340 T4861 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 997344 T4892 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 997346 T4861 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 997349 T4892 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 997349 T4892 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C634 name=collection1 org.apache.solr.core.SolrCore@fbb8544 url=https://127.0.0.1:50540/collection1 node=127.0.0.1:50540_ C634_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:50540, core=collection1, node_name=127.0.0.1:50540_}
[junit4] 2> 997349 T4892 C634 P50540 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50540/collection1/
[junit4] 2> 997349 T4892 C634 P50540 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 997350 T4892 C634 P50540 oasc.SyncStrategy.syncToMe https://127.0.0.1:50540/collection1/ has no replicas
[junit4] 2> 997350 T4892 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50540/collection1/ shard1
[junit4] 2> 997350 T4892 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 997353 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 997359 T4886 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 997364 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 997381 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 997487 T4894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 997488 T4885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 997520 T4892 oasc.ZkController.register We are https://127.0.0.1:50540/collection1/ and leader is https://127.0.0.1:50540/collection1/
[junit4] 2> 997520 T4892 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50540
[junit4] 2> 997520 T4892 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 997520 T4892 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 997521 T4892 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 997524 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 997524 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 997525 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 997525 T4892 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 997531 T4886 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 997532 T4886 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:50540",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:50540_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 997537 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 997642 T4894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 997642 T4885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 998940 T4861 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 998941 T4861 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 998948 T4861 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 998950 T4861 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50544
[junit4] 2> 998953 T4861 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 998954 T4861 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 998954 T4861 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-003
[junit4] 2> 998955 T4861 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-003/'
[junit4] 2> 998995 T4861 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-003/solr.xml
[junit4] 2> 999013 T4861 oasc.CoreContainer.<init> New CoreContainer 472119102
[junit4] 2> 999014 T4861 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-003/]
[junit4] 2> 999014 T4861 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 999016 T4861 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 999019 T4861 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 999019 T4861 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 999020 T4861 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 999020 T4861 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 999020 T4861 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 999021 T4861 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 999021 T4861 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 999026 T4861 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 999027 T4861 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 999027 T4861 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 999028 T4861 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50536/solr
[junit4] 2> 999028 T4861 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 999030 T4861 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 999033 T4905 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@67aa9d57 name:ZooKeeperConnection Watcher:127.0.0.1:50536 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 999034 T4861 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 999037 T4861 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 999041 T4907 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69f4a04d name:ZooKeeperConnection Watcher:127.0.0.1:50536/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 999042 T4861 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 999058 T4861 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1000070 T4861 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50544_
[junit4] 2> 1000073 T4861 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50544_
[junit4] 2> 1000095 T4908 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1000096 T4908 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1000098 T4908 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1000099 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1000099 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1000099 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1000102 T4886 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1000104 T4886 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:50544",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:50544_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1000104 T4886 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 1000104 T4886 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1000110 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1000113 T4907 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1000115 T4894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1000116 T4885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1001100 T4908 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1001100 T4908 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1001102 T4908 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1001102 T4908 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1001105 T4908 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1001105 T4908 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-003/collection1/'
[junit4] 2> 1001106 T4908 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 1001106 T4908 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 1001107 T4908 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 1001153 T4908 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 1001163 T4908 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1001166 T4908 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1001172 T4908 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1001287 T4908 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1001289 T4908 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1001290 T4908 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1001295 T4908 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1001343 T4908 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1001343 T4908 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1001344 T4908 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty1/
[junit4] 2> 1001344 T4908 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6994cbf
[junit4] 2> 1001345 T4908 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty1
[junit4] 2> 1001347 T4908 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty1/index/
[junit4] 2> 1001348 T4908 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 1001350 T4908 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty1/index
[junit4] 2> 1001350 T4908 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=57.4130859375, floorSegmentMB=0.9599609375, forceMergeDeletesPctAllowed=29.834462607382324, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 1001354 T4908 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1001354 T4908 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1001360 T4908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1001360 T4908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1001361 T4908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1001362 T4908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1001362 T4908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1001364 T4908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1001365 T4908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1001366 T4908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1001366 T4908 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1001366 T4908 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1001367 T4908 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1001367 T4908 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1001367 T4908 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1001368 T4908 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1001368 T4908 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1001368 T4908 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1001369 T4908 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 1001380 T4908 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1001383 T4908 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1001383 T4908 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1001384 T4908 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=84.185546875, floorSegmentMB=1.3896484375, forceMergeDeletesPctAllowed=12.252324090316794, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4486034771152828
[junit4] 2> 1001386 T4908 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1001386 T4908 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1001387 T4908 oass.SolrIndexSearcher.<init> Opening Searcher@626bef01[collection1] main
[junit4] 2> 1001387 T4908 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1001390 T4908 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1001391 T4908 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1001392 T4908 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1001393 T4908 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1001393 T4908 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1001395 T4908 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 1001395 T4908 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1001395 T4908 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 1001395 T4908 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1001399 T4909 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@626bef01[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1001402 T4908 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1001404 T4912 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50544 collection:collection1 shard:shard2
[junit4] 2> 1001405 T4861 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 1001405 T4861 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1001406 T4912 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1001419 T4912 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1001424 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1001425 T4912 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1001426 T4912 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C635 name=collection1 org.apache.solr.core.SolrCore@437ffe90 url=https://127.0.0.1:50544/collection1 node=127.0.0.1:50544_ C635_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:50544, core=collection1, node_name=127.0.0.1:50544_}
[junit4] 2> 1001426 T4912 C635 P50544 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50544/collection1/
[junit4] 2> 1001426 T4912 C635 P50544 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1001426 T4912 C635 P50544 oasc.SyncStrategy.syncToMe https://127.0.0.1:50544/collection1/ has no replicas
[junit4] 2> 1001427 T4912 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50544/collection1/ shard2
[junit4] 2> 1001428 T4912 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1001435 T4886 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1001443 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1001447 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1001454 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1001559 T4894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1001560 T4907 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1001561 T4885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1001604 T4912 oasc.ZkController.register We are https://127.0.0.1:50544/collection1/ and leader is https://127.0.0.1:50544/collection1/
[junit4] 2> 1001604 T4912 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50544
[junit4] 2> 1001605 T4912 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1001605 T4912 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 1001605 T4912 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1001609 T4912 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1001609 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1001610 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1001610 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1001614 T4886 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1001615 T4886 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:50544",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:50544_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1001621 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1001725 T4907 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1001726 T4894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1001726 T4885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1002852 T4861 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 1002853 T4861 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1002861 T4861 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1002863 T4861 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50547
[junit4] 2> 1002866 T4861 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1002866 T4861 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1002866 T4861 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-004
[junit4] 2> 1002867 T4861 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-004/'
[junit4] 2> 1002903 T4861 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-004/solr.xml
[junit4] 2> 1002921 T4861 oasc.CoreContainer.<init> New CoreContainer 305562498
[junit4] 2> 1002922 T4861 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-004/]
[junit4] 2> 1002922 T4861 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1002922 T4861 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1002923 T4861 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1002923 T4861 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1002923 T4861 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1002923 T4861 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1002924 T4861 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1002925 T4861 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1002925 T4861 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1002934 T4861 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1002935 T4861 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1002935 T4861 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1002935 T4861 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50536/solr
[junit4] 2> 1002936 T4861 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1002937 T4861 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1002942 T4923 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@eee2443 name:ZooKeeperConnection Watcher:127.0.0.1:50536 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1002943 T4861 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1002949 T4861 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1002952 T4925 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c0d69ae name:ZooKeeperConnection Watcher:127.0.0.1:50536/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1002952 T4861 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1002975 T4861 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1003985 T4861 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50547_
[junit4] 2> 1003987 T4861 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50547_
[junit4] 2> 1004007 T4926 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1004008 T4926 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1004010 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1004010 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1004010 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1004010 T4926 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1004014 T4886 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1004016 T4886 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:50547",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:50547_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1004016 T4886 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1004016 T4886 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1004021 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1004023 T4925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1004023 T4894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1004023 T4885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1004023 T4907 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1005012 T4926 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1005012 T4926 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1005014 T4926 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1005014 T4926 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1005017 T4926 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1005017 T4926 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-004/collection1/'
[junit4] 2> 1005019 T4926 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 1005019 T4926 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 1005019 T4926 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 1005069 T4926 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 1005081 T4926 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1005085 T4926 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1005090 T4926 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1005216 T4926 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1005217 T4926 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1005218 T4926 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1005224 T4926 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1005287 T4926 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1005288 T4926 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1005288 T4926 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty2/
[junit4] 2> 1005288 T4926 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6994cbf
[junit4] 2> 1005289 T4926 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty2
[junit4] 2> 1005290 T4926 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty2/index/
[junit4] 2> 1005291 T4926 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 1005291 T4926 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty2/index
[junit4] 2> 1005293 T4926 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=57.4130859375, floorSegmentMB=0.9599609375, forceMergeDeletesPctAllowed=29.834462607382324, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 1005297 T4926 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1005299 T4926 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1005303 T4926 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1005304 T4926 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1005304 T4926 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1005305 T4926 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1005305 T4926 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1005306 T4926 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1005306 T4926 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1005306 T4926 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1005306 T4926 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1005307 T4926 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1005307 T4926 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1005308 T4926 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1005308 T4926 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1005308 T4926 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1005309 T4926 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1005309 T4926 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1005310 T4926 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 1005319 T4926 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1005320 T4926 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1005321 T4926 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1005322 T4926 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=84.185546875, floorSegmentMB=1.3896484375, forceMergeDeletesPctAllowed=12.252324090316794, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4486034771152828
[junit4] 2> 1005324 T4926 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1005324 T4926 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1005325 T4926 oass.SolrIndexSearcher.<init> Opening Searcher@476daba5[collection1] main
[junit4] 2> 1005326 T4926 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1005329 T4926 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1005330 T4926 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1005331 T4926 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1005332 T4926 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1005332 T4926 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1005333 T4926 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 1005334 T4926 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1005334 T4926 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 1005334 T4926 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1005339 T4927 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@476daba5[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1005340 T4926 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1005342 T4930 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50547 collection:collection1 shard:shard3
[junit4] 2> 1005343 T4861 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 1005344 T4861 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1005345 T4930 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 1005363 T4930 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 1005372 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1005376 T4930 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1005376 T4930 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C636 name=collection1 org.apache.solr.core.SolrCore@56516585 url=https://127.0.0.1:50547/collection1 node=127.0.0.1:50547_ C636_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:50547, core=collection1, node_name=127.0.0.1:50547_}
[junit4] 2> 1005376 T4930 C636 P50547 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50547/collection1/
[junit4] 2> 1005377 T4930 C636 P50547 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1005377 T4930 C636 P50547 oasc.SyncStrategy.syncToMe https://127.0.0.1:50547/collection1/ has no replicas
[junit4] 2> 1005378 T4930 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50547/collection1/ shard3
[junit4] 2> 1005378 T4930 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 1005381 T4886 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1005398 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1005404 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1005415 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1005524 T4907 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1005525 T4885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1005527 T4894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1005527 T4925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1005563 T4930 oasc.ZkController.register We are https://127.0.0.1:50547/collection1/ and leader is https://127.0.0.1:50547/collection1/
[junit4] 2> 1005563 T4930 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50547
[junit4] 2> 1005563 T4930 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1005563 T4930 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 1005564 T4930 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1005565 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1005566 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1005566 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1005567 T4930 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1005571 T4886 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1005572 T4886 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:50547",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:50547_",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 1005577 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1005682 T4894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1005682 T4925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1005682 T4907 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1005682 T4885 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1006915 T4861 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 1006916 T4861 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1006923 T4861 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1006924 T4861 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50550
[junit4] 2> 1006927 T4861 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1006927 T4861 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1006928 T4861 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-005
[junit4] 2> 1006928 T4861 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-005/'
[junit4] 2> 1006963 T4861 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-005/solr.xml
[junit4] 2> 1006980 T4861 oasc.CoreContainer.<init> New CoreContainer 223035801
[junit4] 2> 1006980 T4861 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-8E71F0B3A8B73DF7-001/tempDir-005/]
[junit4] 2> 1006980 T4861 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1006981 T4861 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1006981 T4861 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1006981 T4861 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1006982 T4861 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1006982 T4861 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1006983 T4861 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1006983 T4861 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1006983 T4861 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1006987 T4861 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1006987 T4861 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1006988 T4861 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1006988 T4861 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50536/solr
[junit4] 2> 1006988 T4861 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1006990 T4861 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1006994 T4941 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f2cca31 name:ZooKeeperConnection Watcher:127.0.0.1:50536 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1006995 T4861 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1007000 T4861 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1007003 T4943 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a6becfd name:ZooKeeperConnection Watcher:127.0.0.1:50536/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1007003 T4861 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1007022 T4861 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1008030 T4861 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50550_
[junit4] 2> 1008032 T4861 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50550_
[junit4] 2> 1008050 T4944 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1008050 T4944 oasc.ZkController.publish numShards
[...truncated too long message...]
sc.CachingDirectoryFactory.closeCacheValue looking to close ./solr.cloud.BasicDistributedZkTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty4 [CachedDir<<refCount=0;path=./solr.cloud.BasicDistributedZkTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty4;done=false>>]
[junit4] 2> 1455758 T6591 oasc.CachingDirectoryFactory.close Closing directory: ./solr.cloud.BasicDistributedZkTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty4
[junit4] 2> 1455759 T6591 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./solr.cloud.BasicDistributedZkTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty4/index [CachedDir<<refCount=0;path=./solr.cloud.BasicDistributedZkTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty4/index;done=false>>]
[junit4] 2> 1455759 T6591 oasc.CachingDirectoryFactory.close Closing directory: ./solr.cloud.BasicDistributedZkTest-8E71F0B3A8B73DF7-001/tempDir-001/jetty4/index
[junit4] 2> 1455759 T6591 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 1455760 T6591 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.BasicDistributedZkTest-8E71F0B3A8B73DF7-001
[junit4] 2> NOTE: test params are: codec=Lucene46: {other_tl1=Pulsing41(freqCutoff=4 minBlockSize=10 maxBlockSize=112), SubjectTerms_mfacet=PostingsFormat(name=SimpleText), n_l1=Pulsing41(freqCutoff=4 minBlockSize=10 maxBlockSize=112), multiDefault=Pulsing41(freqCutoff=4 minBlockSize=10 maxBlockSize=112), intDefault=Pulsing41(freqCutoff=4 minBlockSize=10 maxBlockSize=112), n_td1=PostingsFormat(name=SimpleText), n_d1=Pulsing41(freqCutoff=4 minBlockSize=10 maxBlockSize=112), range_facet_l=Pulsing41(freqCutoff=4 minBlockSize=10 maxBlockSize=112), n_f1=PostingsFormat(name=SimpleText), n_tl1=PostingsFormat(name=SimpleText), oddField_s=FST41, n_tf1=Pulsing41(freqCutoff=4 minBlockSize=10 maxBlockSize=112), a_t=Pulsing41(freqCutoff=4 minBlockSize=10 maxBlockSize=112), id=FST41, text=PostingsFormat(name=Lucene41WithOrds), timestamp=Pulsing41(freqCutoff=4 minBlockSize=10 maxBlockSize=112), foo_i=PostingsFormat(name=SimpleText), n_dt1=PostingsFormat(name=SimpleText), a_i1=Pulsing41(freqCutoff=4 minBlockSize=10 maxBlockSize=112), n_ti1=FST41, rnd_b=FST41, _version_=PostingsFormat(name=SimpleText), foo_b=PostingsFormat(name=Lucene41WithOrds), n_tdt1=PostingsFormat(name=SimpleText), foo_d=FST41, foo_f=PostingsFormat(name=Lucene41WithOrds)}, docValues:{timestamp=DocValuesFormat(name=Memory)}, sim=DefaultSimilarity, locale=no_NO_NY, timezone=Asia/Yerevan
[junit4] 2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=2,threads=1,free=223270288,total=362807296
[junit4] 2> NOTE: All tests run in this JVM: [TestHighlightDedupGrouping, OverseerTest, TestInitQParser, OutputWriterTest, ModifyConfFileTest, TestWriterPerf, AlternateDirectoryTest, AnalysisErrorHandlingTest, TestFastOutputStream, TestStressReorder, TestDynamicFieldCollectionResource, TestLRUCache, TestManagedResourceStorage, ClusterStateTest, TestCloudManagedSchemaAddField, HdfsWriteToMultipleCollectionsTest, AtomicUpdatesTest, TestXIncludeConfig, TestCollectionAPI, TestQueryTypes, DocValuesTest, BasicFunctionalityTest, DisMaxRequestHandlerTest, TestCollationFieldDocValues, ConvertedLegacyTest, SOLR749Test, TestFieldCollectionResource, TestCodecSupport, TestSolrXmlPersistor, TestBinaryField, ClusterStateUpdateTest, SolrInfoMBeanTest, TestElisionMultitermQuery, TestSort, ReturnFieldsTest, TestCollapseQParserPlugin, TestReversedWildcardFilterFactory, AutoCommitTest, TestSchemaNameResource, SimplePostToolTest, BlockDirectoryTest, DistributedSpellCheckComponentTest, PreAnalyzedFieldTest, JSONWriterTest, TestSchemaVersionResource, TestAtomicUpdateErrorCases, ZkCLITest, PreAnalyzedUpdateProcessorTest, HdfsCollectionsAPIDistributedZkTest, TestNRTOpen, TestSolrXMLSerializer, CurrencyFieldXmlFileTest, TestLazyCores, DistributedTermsComponentTest, AddSchemaFieldsUpdateProcessorFactoryTest, URLClassifyProcessorTest, TestOmitPositions, TestRTGBase, TestModifyConfFiles, UUIDFieldTest, SimpleFacetsTest, SolrPluginUtilsTest, AbstractAnalyticsFacetTest, DeleteShardTest, TestJmxMonitoredMap, DocValuesMissingTest, TestDocBasedVersionConstraints, UpdateParamsTest, BlockCacheTest, SolrCmdDistributorTest, TestPostingsSolrHighlighter, TestDistributedMissingSort, SuggesterTest, ZkSolrClientTest, TestOverriddenPrefixQueryForCustomFieldType, TestAnalyzedSuggestions, TestDistribDocBasedVersion, TestManagedStopFilterFactory, TestRandomFaceting, TestSearchPerf, DistributedQueryComponentCustomSortTest, FullSolrCloudDistribCmdsTest, TestSearcherReuse, CustomCollectionTest, AbstractAnalyticsStatsTest, DirectUpdateHandlerOptimizeTest, CurrencyFieldOpenExchangeTest, TestPHPSerializedResponseWriter, PathHierarchyTokenizerFactoryTest, TestCloudInspectUtil, TestUpdate, CoreAdminCreateDiscoverTest, IndexSchemaRuntimeFieldTest, ZkControllerTest, UnloadDistributedZkTest, PrimitiveFieldTypeTest, TestMultiCoreConfBootstrap, TestConfig, BadCopyFieldTest, TestManagedResource, TestFieldResource, CacheHeaderTest, TestSolrIndexConfig, TestLuceneMatchVersion, LoggingHandlerTest, TestShortCircuitedRequests, TestCSVLoader, DeleteInactiveReplicaTest, TestMergePolicyConfig, QueryParsingTest, FieldAnalysisRequestHandlerTest, TestFieldTypeResource, XsltUpdateRequestHandlerTest, DistributedQueryElevationComponentTest, DOMUtilTest, TestStandardQParsers, CursorMarkTest, DateFieldTest, DocumentAnalysisRequestHandlerTest, CopyFieldTest, OverseerCollectionProcessorTest, ConnectionManagerTest, TestSurroundQueryParser, TestFileDictionaryLookup, OverseerRolesTest, TestComponentsName, ExternalFileFieldSortTest, HighlighterConfigTest, ResourceLoaderTest, TestCollationField, TestRestManager, IndexSchemaTest, CoreAdminRequestStatusTest, TestBlendedInfixSuggestions, SpellingQueryConverterTest, TestExpandComponent, XmlUpdateRequestHandlerTest, QueryFacetTest, ShardRoutingCustomTest, TestRandomMergePolicy, SSLMigrationTest, TestFunctionQuery, AliasIntegrationTest, SchemaVersionSpecificBehaviorTest, RemoteQueryErrorTest, IndexBasedSpellCheckerTest, InfoHandlerTest, ChaosMonkeySafeLeaderTest, TestDistributedSearch, TestDefaultSearchFieldResource, ChangedSchemaMergeTest, TestSolrQueryParserResource, HdfsLockFactoryTest, HdfsRecoveryZkTest, DistribCursorPagingTest, StressHdfsTest, TestFreeTextSuggestions, NotRequiredUniqueKeyTest, OverseerStatusTest, TestSuggestSpellingConverter, SoftAutoCommitTest, ShardSplitTest, TestDocumentBuilder, SolrTestCaseJ4Test, TestBinaryResponseWriter, TestZkChroot, DefaultValueUpdateProcessorTest, TestDefaultSimilarityFactory, MinimalSchemaTest, HighlighterTest, TestSolrCoreProperties, SortByFunctionTest, TestDynamicFieldResource, TestSolrDeletionPolicy2, SystemInfoHandlerTest, ResponseLogComponentTest, CircularListTest, TestSolrDeletionPolicy1, UniqFieldsUpdateProcessorFactoryTest, LeaderElectionIntegrationTest, TestFiltering, ShardRoutingTest, TestSolrXml, LukeRequestHandlerTest, TestFastLRUCache, HdfsBasicDistributedZkTest, PingRequestHandlerTest, TestIndexSearcher, BasicZkTest, EnumFieldTest, TestInfoStreamLogging, TestCharFilters, BasicDistributedZkTest]
[junit4] Completed in 31.99s, 1 test, 1 failure <<< FAILURES!
[...truncated 626 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1296: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:920: There were test failures: 398 suites, 1673 tests, 2 failures, 42 ignored (15 assumptions)
Total time: 85 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1563
- Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1563/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseG1GC
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 102 from http://127.0.0.1:51324/collection1lastClient and got 94 from http://127.0.0.1:51333/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 102 from http://127.0.0.1:51324/collection1lastClient and got 94 from http://127.0.0.1:51333/collection1
at __randomizedtesting.SeedInfo.seed([E7C2A15175B9AA9D:66242F4902E6CAA1]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1130)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1109)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11176 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/init-core-data-001
[junit4] 2> 2034278 T6715 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 2034279 T6715 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 2034294 T6715 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 2034295 T6715 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2034298 T6716 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 2034399 T6715 oasc.ZkTestServer.run start zk server on port:51317
[junit4] 2> 2034401 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2034411 T6722 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74d50c0b name:ZooKeeperConnection Watcher:127.0.0.1:51317 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2034412 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2034413 T6715 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 2034430 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2034437 T6724 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a100f8b name:ZooKeeperConnection Watcher:127.0.0.1:51317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2034439 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2034440 T6715 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 2034451 T6715 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 2034459 T6715 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 2034465 T6715 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 2034474 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 2034475 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 2034490 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 2034491 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 2034498 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 2034498 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 2034505 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 2034505 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 2034512 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 2034513 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 2034520 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 2034521 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 2034527 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 2034527 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 2034537 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 2034537 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 2034546 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 2034547 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 2034557 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 2034558 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 2034567 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 2034568 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 2036217 T6715 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 2036223 T6715 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
[junit4] 2> 2036224 T6715 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 2036225 T6715 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 2036225 T6715 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002
[junit4] 2> 2036226 T6715 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002/'
[junit4] 2> 2036294 T6715 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002/solr.xml
[junit4] 2> 2036617 T6715 oasc.CoreContainer.<init> New CoreContainer 1161568699
[junit4] 2> 2036624 T6715 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002/]
[junit4] 2> 2036638 T6715 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 2036640 T6715 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 2036646 T6715 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 2036647 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 2036649 T6715 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 2036651 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 2036657 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 2036658 T6715 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 2036660 T6715 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 2036672 T6715 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 2036675 T6715 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 2036679 T6715 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 2036683 T6715 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51317/solr
[junit4] 2> 2036685 T6715 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 2036710 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2036718 T6735 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6eca9f7f name:ZooKeeperConnection Watcher:127.0.0.1:51317 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2036718 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2036724 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2036728 T6737 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e771953 name:ZooKeeperConnection Watcher:127.0.0.1:51317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2036729 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2036733 T6715 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 2036743 T6715 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 2036753 T6715 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 2036763 T6715 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 2036774 T6715 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 2036786 T6715 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 2036791 T6715 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51320_
[junit4] 2> 2036794 T6715 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51320_
[junit4] 2> 2036805 T6715 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 2036814 T6715 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 2036827 T6715 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51320_
[junit4] 2> 2036827 T6715 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 2036835 T6715 oasc.Overseer.start Overseer (id=91723437819297795-127.0.0.1:51320_-n_0000000000) starting
[junit4] 2> 2036846 T6715 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 2036884 T6739 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 2036888 T6739 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:51320_
[junit4] 2> 2036889 T6715 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 2036897 T6715 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 2036903 T6715 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 2036912 T6738 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 2036924 T6740 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 2036925 T6740 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2036928 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2036928 T6740 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 2036933 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 2036935 T6738 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:51320",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51320_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 2036936 T6738 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 2036936 T6738 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 2036943 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2036946 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2037930 T6740 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 2037930 T6740 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 2037932 T6740 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 2037933 T6740 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 2037936 T6740 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 2037936 T6740 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002/collection1/'
[junit4] 2> 2037938 T6740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 2037938 T6740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 2037940 T6740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 2038071 T6740 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 2038217 T6740 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 2038221 T6740 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 2038237 T6740 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 2039080 T6740 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 2039086 T6740 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 2039090 T6740 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 2039100 T6740 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 2039197 T6740 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 2039197 T6740 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 2039199 T6740 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/control/data/
[junit4] 2> 2039199 T6740 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@57a9514b
[junit4] 2> 2039201 T6740 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/control/data
[junit4] 2> 2039203 T6740 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/control/data/index/
[junit4] 2> 2039204 T6740 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 2039205 T6740 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/control/data/index
[junit4] 2> 2039205 T6740 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=54.892578125, floorSegmentMB=1.830078125, forceMergeDeletesPctAllowed=3.2255022366820043, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 2039212 T6740 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 2039212 T6740 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 2039223 T6740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 2039224 T6740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 2039225 T6740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 2039225 T6740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 2039225 T6740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 2039226 T6740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 2039227 T6740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2039227 T6740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2039227 T6740 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 2039228 T6740 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 2039229 T6740 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 2039229 T6740 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 2039230 T6740 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 2039231 T6740 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 2039232 T6740 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 2039232 T6740 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 2039234 T6740 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 2039248 T6740 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2039254 T6740 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 2039254 T6740 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 2039256 T6740 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=27, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 2039259 T6740 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 2039260 T6740 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 2039261 T6740 oass.SolrIndexSearcher.<init> Opening Searcher@723c43a9[collection1] main
[junit4] 2> 2039261 T6740 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 2039269 T6740 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 2039270 T6740 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 2039271 T6740 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2039272 T6740 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 2039272 T6740 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2039274 T6740 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 2039276 T6740 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2039276 T6740 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 2039276 T6740 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 2039286 T6741 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@723c43a9[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 2039289 T6740 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 2039292 T6744 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51320 collection:control_collection shard:shard1
[junit4] 2> 2039293 T6715 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 2039294 T6715 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 2039294 T6744 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 2039298 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2039304 T6746 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2308dacb name:ZooKeeperConnection Watcher:127.0.0.1:51317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2039305 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2039309 T6715 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 2039317 T6715 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 2039320 T6744 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 2039326 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2039330 T6744 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 2039330 T6744 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1008 name=collection1 org.apache.solr.core.SolrCore@502de817 url=http://127.0.0.1:51320/collection1 node=127.0.0.1:51320_ C1008_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:51320, core=collection1, node_name=127.0.0.1:51320_}
[junit4] 2> 2039331 T6744 C1008 P51320 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51320/collection1/
[junit4] 2> 2039331 T6744 C1008 P51320 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 2039332 T6744 C1008 P51320 oasc.SyncStrategy.syncToMe http://127.0.0.1:51320/collection1/ has no replicas
[junit4] 2> 2039332 T6744 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51320/collection1/ shard1
[junit4] 2> 2039332 T6744 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 2039333 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 2039350 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2039354 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2039355 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2039377 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 2039388 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2039495 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2039495 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2039528 T6744 oasc.ZkController.register We are http://127.0.0.1:51320/collection1/ and leader is http://127.0.0.1:51320/collection1/
[junit4] 2> 2039529 T6744 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51320
[junit4] 2> 2039530 T6744 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 2039530 T6744 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 2039531 T6744 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2039537 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2039537 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2039539 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2039542 T6744 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 2039547 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 2039548 T6738 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:51320",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51320_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 2039558 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2039667 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2039669 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2041187 T6715 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 2041189 T6715 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 2041196 T6715 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
[junit4] 2> 2041196 T6715 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 2041197 T6715 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 2041197 T6715 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003
[junit4] 2> 2041197 T6715 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003/'
[junit4] 2> 2041258 T6715 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003/solr.xml
[junit4] 2> 2041375 T6715 oasc.CoreContainer.<init> New CoreContainer 1081531971
[junit4] 2> 2041376 T6715 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003/]
[junit4] 2> 2041377 T6715 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 2041378 T6715 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 2041378 T6715 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 2041378 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 2041379 T6715 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 2041379 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 2041380 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 2041380 T6715 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 2041380 T6715 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 2041381 T6715 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 2041382 T6715 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 2041383 T6715 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 2041383 T6715 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51317/solr
[junit4] 2> 2041384 T6715 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 2041385 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2041391 T6757 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3bfa21c5 name:ZooKeeperConnection Watcher:127.0.0.1:51317 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2041391 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2041397 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2041400 T6759 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@594dcf55 name:ZooKeeperConnection Watcher:127.0.0.1:51317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2041400 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2041425 T6715 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 2042435 T6715 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51324_
[junit4] 2> 2042438 T6715 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51324_
[junit4] 2> 2042477 T6760 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 2042477 T6760 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2042480 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2042480 T6760 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 2042481 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2042481 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2042485 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 2042487 T6738 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:51324",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51324_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 2042487 T6738 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 2042487 T6738 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 2042494 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2042496 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2042497 T6759 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2042497 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2043482 T6760 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 2043482 T6760 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 2043484 T6760 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 2043484 T6760 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 2043487 T6760 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 2043487 T6760 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003/collection1/'
[junit4] 2> 2043489 T6760 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 2043490 T6760 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 2043491 T6760 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 2043594 T6760 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 2043744 T6760 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 2043747 T6760 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 2043758 T6760 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 2044419 T6760 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 2044423 T6760 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 2044425 T6760 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 2044433 T6760 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 2044492 T6760 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 2044492 T6760 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 2044494 T6760 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty1/
[junit4] 2> 2044494 T6760 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@57a9514b
[junit4] 2> 2044496 T6760 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty1
[junit4] 2> 2044497 T6760 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty1/index/
[junit4] 2> 2044497 T6760 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 2044498 T6760 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty1/index
[junit4] 2> 2044498 T6760 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=54.892578125, floorSegmentMB=1.830078125, forceMergeDeletesPctAllowed=3.2255022366820043, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 2044504 T6760 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 2044504 T6760 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 2044514 T6760 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 2044514 T6760 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 2044515 T6760 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 2044515 T6760 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 2044515 T6760 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 2044515 T6760 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 2044516 T6760 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2044517 T6760 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2044517 T6760 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 2044518 T6760 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 2044519 T6760 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 2044520 T6760 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 2044520 T6760 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 2044521 T6760 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 2044522 T6760 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 2044523 T6760 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 2044524 T6760 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 2044537 T6760 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2044542 T6760 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 2044543 T6760 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 2044544 T6760 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=27, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 2044546 T6760 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 2044546 T6760 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 2044547 T6760 oass.SolrIndexSearcher.<init> Opening Searcher@6ad4dfe1[collection1] main
[junit4] 2> 2044547 T6760 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 2044550 T6760 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 2044550 T6760 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 2044552 T6760 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2044552 T6760 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 2044553 T6760 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2044554 T6760 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 2044555 T6760 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2044555 T6760 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 2044555 T6760 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 2044561 T6761 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6ad4dfe1[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 2044565 T6760 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 2044568 T6764 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51324 collection:collection1 shard:shard1
[junit4] 2> 2044569 T6715 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 2044569 T6715 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 2044571 T6764 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 2044582 T6764 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 2044588 T6764 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 2044589 T6764 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1009 name=collection1 org.apache.solr.core.SolrCore@1665b87c url=http://127.0.0.1:51324/collection1 node=127.0.0.1:51324_ C1009_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:51324, core=collection1, node_name=127.0.0.1:51324_}
[junit4] 2> 2044589 T6764 C1009 P51324 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51324/collection1/
[junit4] 2> 2044590 T6764 C1009 P51324 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 2044590 T6764 C1009 P51324 oasc.SyncStrategy.syncToMe http://127.0.0.1:51324/collection1/ has no replicas
[junit4] 2> 2044590 T6764 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51324/collection1/ shard1
[junit4] 2> 2044591 T6764 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 2044595 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2044605 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 2044610 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2044624 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2044625 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2044629 T6759 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2044648 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 2044768 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2044769 T6759 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2044776 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2044821 T6764 oasc.ZkController.register We are http://127.0.0.1:51324/collection1/ and leader is http://127.0.0.1:51324/collection1/
[junit4] 2> 2044822 T6764 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51324
[junit4] 2> 2044822 T6764 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 2044822 T6764 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 2044822 T6764 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2044826 T6764 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 2044828 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2044828 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2044828 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2044832 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 2044834 T6738 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:51324",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51324_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 2044839 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2044946 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2044947 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2044952 T6759 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2046199 T6715 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 2046200 T6715 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 2046206 T6715 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
[junit4] 2> 2046207 T6715 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 2046207 T6715 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 2046208 T6715 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004
[junit4] 2> 2046208 T6715 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004/'
[junit4] 2> 2046264 T6715 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004/solr.xml
[junit4] 2> 2046359 T6715 oasc.CoreContainer.<init> New CoreContainer 1835826604
[junit4] 2> 2046360 T6715 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004/]
[junit4] 2> 2046362 T6715 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 2046362 T6715 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 2046362 T6715 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 2046363 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 2046363 T6715 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 2046363 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 2046364 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 2046364 T6715 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 2046364 T6715 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 2046365 T6715 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 2046366 T6715 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 2046367 T6715 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 2046367 T6715 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51317/solr
[junit4] 2> 2046367 T6715 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 2046369 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2046372 T6775 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6daedf53 name:ZooKeeperConnection Watcher:127.0.0.1:51317 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2046373 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2046378 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2046381 T6777 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b1040d1 name:ZooKeeperConnection Watcher:127.0.0.1:51317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2046381 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2046401 T6715 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 2047411 T6715 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51327_
[junit4] 2> 2047414 T6715 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51327_
[junit4] 2> 2047438 T6778 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 2047439 T6778 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2047441 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2047441 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2047441 T6778 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 2047442 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2047446 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 2047447 T6738 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:51327",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51327_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 2047448 T6738 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 2047448 T6738 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 2047454 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2047456 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2047456 T6759 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2047456 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2047456 T6777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2048444 T6778 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 2048444 T6778 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 2048446 T6778 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 2048446 T6778 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 2048450 T6778 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 2048451 T6778 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004/collection1/'
[junit4] 2> 2048452 T6778 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 2048452 T6778 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 2048453 T6778 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 2048541 T6778 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 2048634 T6778 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 2048637 T6778 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 2048649 T6778 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 2049375 T6778 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 2049383 T6778 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 2049388 T6778 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 2049397 T6778 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 2049457 T6778 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 2049458 T6778 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 2049459 T6778 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty2/
[junit4] 2> 2049459 T6778 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@57a9514b
[junit4] 2> 2049461 T6778 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty2
[junit4] 2> 2049462 T6778 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty2/index/
[junit4] 2> 2049462 T6778 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 2049463 T6778 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty2/index
[junit4] 2> 2049465 T6778 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=54.892578125, floorSegmentMB=1.830078125, forceMergeDeletesPctAllowed=3.2255022366820043, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 2049471 T6778 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 2049472 T6778 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 2049480 T6778 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 2049481 T6778 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 2049482 T6778 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 2049482 T6778 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 2049482 T6778 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 2049482 T6778 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 2049483 T6778 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2049484 T6778 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2049484 T6778 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 2049486 T6778 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 2049486 T6778 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 2049487 T6778 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 2049487 T6778 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 2049488 T6778 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 2049488 T6778 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 2049489 T6778 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 2049490 T6778 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 2049503 T6778 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2049508 T6778 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 2049509 T6778 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 2049510 T6778 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=27, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 2049512 T6778 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 2049512 T6778 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 2049512 T6778 oass.SolrIndexSearcher.<init> Opening Searcher@1852ba3f[collection1] main
[junit4] 2> 2049513 T6778 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 2049516 T6778 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 2049517 T6778 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 2049518 T6778 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2049518 T6778 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 2049519 T6778 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2049521 T6778 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 2049521 T6778 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2049521 T6778 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 2049522 T6778 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 2049530 T6779 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1852ba3f[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 2049533 T6778 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 2049535 T6782 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51327 collection:collection1 shard:shard3
[junit4] 2> 2049536 T6715 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 2049537 T6715 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 2049538 T6782 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 2049548 T6782 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 2049555 T6782 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 2049555 T6782 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1010 name=collection1 org.apache.solr.core.SolrCore@72ae959e url=http://127.0.0.1:51327/collection1 node=127.0.0.1:51327_ C1010_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:51327, core=collection1, node_name=127.0.0.1:51327_}
[junit4] 2> 2049555 T6782 C1010 P51327 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51327/collection1/
[junit4] 2> 2049556 T6782 C1010 P51327 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 2049556 T6782 C1010 P51327 oasc.SyncStrategy.syncToMe http://127.0.0.1:51327/collection1/ has no replicas
[junit4] 2> 2049556 T6782 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51327/collection1/ shard3
[junit4] 2> 2049556 T6782 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 2049558 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2049564 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 2049574 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2049577 T6759 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2049577 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2049578 T6777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2049578 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2049591 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 2049599 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2049707 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2049707 T6759 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2049707 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2049707 T6777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2049738 T6782 oasc.ZkController.register We are http://127.0.0.1:51327/collection1/ and leader is http://127.0.0.1:51327/collection1/
[junit4] 2> 2049739 T6782 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51327
[junit4] 2> 2049739 T6782 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 2049740 T6782 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 2049740 T6782 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2049745 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2049746 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2049746 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2049748 T6782 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 2049752 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 2049753 T6738 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:51327",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51327_",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 2049760 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2049865 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2049866 T6777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2049865 T6759 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2049866 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2051208 T6715 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 2051209 T6715 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 2051214 T6715 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
[junit4] 2> 2051215 T6715 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 2051215 T6715 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 2051216 T6715 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-005
[junit4] 2> 2051216 T6715 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-005/'
[junit4] 2> 2051277 T6715 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-005/solr.xml
[junit4] 2> 2051377 T6715 oasc.CoreContainer.<init> New CoreContainer 2009942148
[junit4] 2> 2051377 T6715 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-005/]
[junit4] 2> 2051379 T6715 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 2051379 T6715 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 2051380 T6715 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 2051380 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 2051380 T6715 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 2051381 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 2051381 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 2051381 T6715 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 2051381 T6715 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 2051382 T6715 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 2051383 T6715 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 2051383 T6715 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 2051383 T6715 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51317/solr
[junit4] 2> 2051384 T6715 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 2051386 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2051389 T6793 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46540f8e name:ZooKeeperConnection Watcher:127.0.0.1:51317 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2051390 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2051394 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2051397 T6795 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9b7bf39 name:ZooKeeperConnection Watcher:127.0.0.1:51317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2051397 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2051416 T6715 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 2052426 T6715 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51330_
[junit4] 2> 2052428 T6715 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51330_
[junit4] 2> 2052453 T6796 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 2052454 T6796 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2052456 T6796 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 2052456 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2052457 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2052457 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2052460 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 2052462 T6738 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:51330",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51330_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 2052462 T6738 oasc.Overseer$ClusterStateUpdater.updateState Collection already exi
[...truncated too long message...]
tate Closing SolrCoreState
[junit4] 2> 2149040 T6959 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 2149040 T6959 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 2149064 T6959 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 2149066 T6959 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 2149066 T6959 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty11 [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty11;done=false>>]
[junit4] 2> 2149066 T6959 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty11
[junit4] 2> 2149067 T6959 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty11/index [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty11/index;done=false>>]
[junit4] 2> 2149067 T6959 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty11/index
[junit4] 2> 2149067 T6959 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 2149068 T6959 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001
[junit4] 2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=461), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=461)), sim=DefaultSimilarity, locale=hi_IN, timezone=Africa/Sao_Tome
[junit4] 2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_55 (64-bit)/cpus=2,threads=1,free=379610632,total=534773760
[junit4] 2> NOTE: All tests run in this JVM: [TestPseudoReturnFields, TestDistributedSearch, SynonymTokenizerTest, DirectUpdateHandlerTest, TestCollectionAPI, TestComplexPhraseQParserPlugin, StatelessScriptUpdateProcessorFactoryTest, HdfsSyncSliceTest, NotRequiredUniqueKeyTest, EnumFieldTest, TestDynamicFieldCollectionResource, TestSolrXmlPersistor, TestQueryUtils, HdfsCollectionsAPIDistributedZkTest, TestSolrXMLSerializer, TestConfigSets, PreAnalyzedUpdateProcessorTest, TestSearcherReuse, TestPHPSerializedResponseWriter, TestPhraseSuggestions, TestBinaryField, TestInitQParser, TestLMDirichletSimilarityFactory, TestReplicationHandler, HttpPartitionTest, TestShortCircuitedRequests, PrimitiveFieldTypeTest, DistributedExpandComponentTest, TestDocBasedVersionConstraints, TestJmxMonitoredMap, CursorMarkTest, TestSchemaSimilarityResource, QueryElevationComponentTest, JsonLoaderTest, LoggingHandlerTest, ZkNodePropsTest, TestSearchPerf, OverseerRolesTest, TestDistribDocBasedVersion, MoreLikeThisHandlerTest, FieldAnalysisRequestHandlerTest, TestInfoStreamLogging, RangeFacetTest, TestRealTimeGet, SliceStateUpdateTest, StatsComponentTest, TestTrie, BadCopyFieldTest, TestCollationFieldDocValues, LeaderElectionIntegrationTest, TestModifyConfFiles, TestRTGBase, SolrCoreTest, DisMaxRequestHandlerTest, ClusterStateUpdateTest, ChangedSchemaMergeTest, LegacyHTMLStripCharFilterTest, DistributedQueryComponentCustomSortTest, ChaosMonkeyNothingIsSafeTest, ConvertedLegacyTest, LeaderElectionTest, BasicDistributedZk2Test, TestCloudInspectUtil, TestImplicitCoreProperties, AbstractAnalyticsStatsTest, TestFoldingMultitermQuery, ExpressionTest, CacheHeaderTest, SearchHandlerTest, SolrPluginUtilsTest, TestSolr4Spatial, TestRestManager, NoFacetTest, TestDefaultSearchFieldResource, TestUtils, SampleTest, TestLazyCores, PingRequestHandlerTest, SSLMigrationTest, TestManagedSynonymFilterFactory, TestOverriddenPrefixQueryForCustomFieldType, SuggestComponentTest, PreAnalyzedFieldTest, RemoteQueryErrorTest, BadComponentTest, RegexBoostProcessorTest, TestSort, TestSolrDeletionPolicy2, WordBreakSolrSpellCheckerTest, TestElisionMultitermQuery, TestSchemaResource, TestPostingsSolrHighlighter, FunctionTest, BasicDistributedZkTest, TestCollapseQParserPlugin, TestOmitPositions, TestShardHandlerFactory, TestNRTOpen, TestZkChroot, TestFastOutputStream, SuggesterTest, TestIndexSearcher, ShardRoutingCustomTest, TestSolrXml, ParsingFieldUpdateProcessorsTest, AbstractAnalyticsFacetTest, TestSolrJ, SuggesterFSTTest, TestQuerySenderNoQuery, TestAddFieldRealTimeGet, TestFieldResource, TestSchemaVersionResource, ExternalFileFieldSortTest, FieldFacetTest, DateMathParserTest, CustomCollectionTest, SchemaVersionSpecificBehaviorTest, UpdateParamsTest, TestDistributedGrouping, MBeansHandlerTest, TestQuerySenderListener, TestFaceting, UpdateRequestProcessorFactoryTest, PathHierarchyTokenizerFactoryTest, XsltUpdateRequestHandlerTest, QueryFacetTest, SuggesterWFSTTest, RequestHandlersTest, TestMultiCoreConfBootstrap, CurrencyFieldXmlFileTest, TestDynamicFieldResource, SoftAutoCommitTest, SpellCheckCollatorTest, TestJmxIntegration, TestDefaultSimilarityFactory, ConnectionManagerTest, TestBadConfig, DocExpirationUpdateProcessorFactoryTest, CursorPagingTest, DebugComponentTest, SolrRequestParserTest, TestSimpleQParserPlugin, SyncSliceTest, OverseerCollectionProcessorTest, TestCursorMarkWithoutUniqueKey, BasicFunctionalityTest, ModifyConfFileTest, HdfsBasicDistributedZkTest, TermsComponentTest, HdfsRecoveryZkTest, OpenCloseCoreStressTest, TestFieldTypeResource, UniqFieldsUpdateProcessorFactoryTest, TestFastWriter, OverseerTest, AddBlockUpdateTest, TestComponentsName, TestExceedMaxTermLength, DistanceFunctionTest, DocumentBuilderTest, QueryResultKeyTest, TestSolrDeletionPolicy1, DistributedSpellCheckComponentTest, SolrTestCaseJ4Test, TestSolrXmlPersistence, TestSuggestSpellingConverter, DistribDocExpirationUpdateProcessorTest, TestManagedResource, ReturnFieldsTest, ZkCLITest, AnalysisAfterCoreReloadTest, TestMergePolicyConfig, TestFieldSortValues, TestReloadAndDeleteDocs, TestRandomMergePolicy, TestCSVLoader, TestFuzzyAnalyzedSuggestions, TestSurroundQueryParser, AlternateDirectoryTest, BlockDirectoryTest, SolrInfoMBeanTest, TestLRUCache, SpellPossibilityIteratorTest, PrimUtilsTest, CoreAdminHandlerTest, StressHdfsTest, TestSolrQueryParserDefaultOperatorResource, TestCopyFieldCollectionResource, SimpleFacetsTest, TestIndexingPerformance, BlockCacheTest, ScriptEngineTest, ChaosMonkeySafeLeaderTest]
[junit4] Completed in 115.85s, 1 test, 1 failure <<< FAILURES!
[...truncated 618 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1296: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:920: There were test failures: 399 suites, 1674 tests, 1 failure, 42 ignored (15 assumptions)
Total time: 117 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.8.0) - Build # 1562
- Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1562/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC
All tests passed
Build Log:
[...truncated 20550 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:92: The following files contain @author tags, tabs or nocommits:
* solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java
Total time: 102 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1561
- Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1561/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
All tests passed
Build Log:
[...truncated 17656 lines...]
[junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-map-reduce/test/temp/junit4-J0-20140508_055916_990.syserr
[junit4] >>> JVM J0: stderr (verbatim) ----
[junit4] 2014-05-08 05:59:30.131 java[559:650b] Unable to load realm info from SCDynamicStore
[junit4] <<< JVM J0: EOF ----
[...truncated 2849 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:92: The following files contain @author tags, tabs or nocommits:
* solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java
Total time: 127 minutes 24 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1560
- Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1560/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseG1GC
All tests passed
Build Log:
[...truncated 17666 lines...]
[junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-map-reduce/test/temp/junit4-J0-20140507_221254_942.syserr
[junit4] >>> JVM J0: stderr (verbatim) ----
[junit4] 2014-05-07 22:13:50.647 java[277:6d03] Unable to load realm info from SCDynamicStore
[junit4] <<< JVM J0: EOF ----
[...truncated 2849 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:92: The following files contain @author tags, tabs or nocommits:
* solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java
Total time: 127 minutes 11 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.8.0) - Build # 1559
- Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1559/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
All tests passed
Build Log:
[...truncated 20421 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:92: The following files contain @author tags, tabs or nocommits:
* solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java
Total time: 103 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure