You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/05/03 19:02:34 UTC
[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.8.0) - Build # 1548
- Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1548/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
1 tests failed.
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
expected:<0> but was:<1>
Stack Trace:
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([75930B0030B69967:F475851847E9F95B]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:141)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11077 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/init-core-data-001
[junit4] 2> 1846890 T5908 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 1846891 T5908 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_v/n
[junit4] 2> 1846897 T5908 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 1846899 T5908 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1846900 T5909 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1847002 T5908 oasc.ZkTestServer.run start zk server on port:51071
[junit4] 2> 1847004 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1847011 T5915 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@36b056ec name:ZooKeeperConnection Watcher:127.0.0.1:51071 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1847012 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1847012 T5908 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1847021 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1847026 T5917 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11cfc762 name:ZooKeeperConnection Watcher:127.0.0.1:51071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1847027 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1847027 T5908 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1847034 T5908 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1847039 T5908 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1847043 T5908 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1847048 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1847049 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1847057 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 1847058 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1847065 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1847065 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1847070 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1847071 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1847077 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1847079 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1847084 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1847084 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1847091 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1847092 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 1847103 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1847103 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1847108 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1847109 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1847116 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1847116 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1847121 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1847122 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1847132 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1847137 T5919 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7433772c name:ZooKeeperConnection Watcher:127.0.0.1:51071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1847137 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1848837 T5908 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1848844 T5908 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1848846 T5908 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:51075
[junit4] 2> 1848850 T5908 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1848851 T5908 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1848851 T5908 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002
[junit4] 2> 1848852 T5908 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002/'
[junit4] 2> 1848899 T5908 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002/solr.xml
[junit4] 2> 1848928 T5908 oasc.CoreContainer.<init> New CoreContainer 2033464285
[junit4] 2> 1848928 T5908 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002/]
[junit4] 2> 1848929 T5908 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1848930 T5908 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1848930 T5908 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1848931 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1848931 T5908 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1848931 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1848931 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1848932 T5908 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1848933 T5908 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1848937 T5908 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1848937 T5908 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1848938 T5908 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1848938 T5908 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51071/solr
[junit4] 2> 1848938 T5908 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1848941 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1848944 T5930 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61d1b192 name:ZooKeeperConnection Watcher:127.0.0.1:51071 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1848945 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1848953 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1848955 T5932 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d184f21 name:ZooKeeperConnection Watcher:127.0.0.1:51071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1848956 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1848959 T5908 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1848967 T5908 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1848979 T5908 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1848986 T5908 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1848997 T5908 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1849008 T5908 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1849011 T5908 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51075__v%2Fn
[junit4] 2> 1849013 T5908 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51075__v%2Fn
[junit4] 2> 1849019 T5908 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1849024 T5908 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1849034 T5908 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51075__v%2Fn
[junit4] 2> 1849034 T5908 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1849037 T5908 oasc.Overseer.start Overseer (id=91693664051200004-127.0.0.1:51075__v%2Fn-n_0000000000) starting
[junit4] 2> 1849044 T5908 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1849068 T5934 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1849072 T5934 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:51075__v%2Fn
[junit4] 2> 1849072 T5908 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1849079 T5908 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1849083 T5908 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1849094 T5933 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1849097 T5935 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 1849097 T5935 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1849099 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1849100 T5935 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1849102 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1849104 T5933 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:51075/_v/n",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51075__v%2Fn",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1849104 T5933 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 1849104 T5933 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1849110 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1849112 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1850105 T5935 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1850106 T5935 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1850111 T5935 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1850112 T5935 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1850116 T5935 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1850116 T5935 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002/collection1/'
[junit4] 2> 1850120 T5935 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 1850121 T5935 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 1850121 T5935 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 1850180 T5935 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 1850201 T5935 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1850205 T5935 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1850209 T5935 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1850376 T5935 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1850377 T5935 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1850378 T5935 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1850383 T5935 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1850435 T5935 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 1850435 T5935 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1850435 T5935 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/control/data/
[junit4] 2> 1850436 T5935 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@fcc0205
[junit4] 2> 1850438 T5935 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/control/data
[junit4] 2> 1850439 T5935 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/control/data/index/
[junit4] 2> 1850440 T5935 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 1850441 T5935 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/control/data/index
[junit4] 2> 1850443 T5935 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=67.3037109375, floorSegmentMB=1.296875, forceMergeDeletesPctAllowed=18.754281475137972, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1850448 T5935 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 1850448 T5935 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1850452 T5935 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1850454 T5935 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1850455 T5935 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1850455 T5935 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1850456 T5935 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1850456 T5935 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1850457 T5935 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1850457 T5935 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1850458 T5935 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1850459 T5935 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1850459 T5935 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1850459 T5935 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1850459 T5935 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1850460 T5935 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1850460 T5935 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1850461 T5935 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1850461 T5935 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 1850470 T5935 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1850473 T5935 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1850473 T5935 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1850475 T5935 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=17, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7277011005546191]
[junit4] 2> 1850488 T5935 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 1850488 T5935 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1850489 T5935 oass.SolrIndexSearcher.<init> Opening Searcher@235db3b4[collection1] main
[junit4] 2> 1850489 T5935 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1850491 T5935 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1850492 T5935 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1850494 T5935 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1850495 T5935 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1850495 T5935 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1850497 T5935 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 1850497 T5935 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1850497 T5935 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 1850498 T5935 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1850500 T5936 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@235db3b4[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1850501 T5935 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1850503 T5939 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:51075/_v/n collection:control_collection shard:shard1
[junit4] 2> 1850506 T5908 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 1850506 T5908 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1850512 T5939 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 1850514 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1850520 T5941 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@363a5978 name:ZooKeeperConnection Watcher:127.0.0.1:51071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1850520 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1850525 T5908 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1850532 T5908 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1850558 T5939 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1850568 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1850569 T5939 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1850569 T5939 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C3551 name=collection1 org.apache.solr.core.SolrCore@31a62f76 url=https://127.0.0.1:51075/_v/n/collection1 node=127.0.0.1:51075__v%2Fn C3551_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:51075/_v/n, core=collection1, node_name=127.0.0.1:51075__v%2Fn}
[junit4] 2> 1850570 T5939 C3551 P51075 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:51075/_v/n/collection1/
[junit4] 2> 1850570 T5939 C3551 P51075 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1850570 T5939 C3551 P51075 oasc.SyncStrategy.syncToMe https://127.0.0.1:51075/_v/n/collection1/ has no replicas
[junit4] 2> 1850571 T5939 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:51075/_v/n/collection1/ shard1
[junit4] 2> 1850571 T5939 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 1850578 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1850594 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1850605 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1850623 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1850628 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1850628 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1850709 T5939 oasc.ZkController.register We are https://127.0.0.1:51075/_v/n/collection1/ and leader is https://127.0.0.1:51075/_v/n/collection1/
[junit4] 2> 1850709 T5939 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:51075/_v/n
[junit4] 2> 1850710 T5939 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1850710 T5939 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 1850710 T5939 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1850716 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1850720 T5939 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1850727 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1850736 T5933 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:51075/_v/n",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51075__v%2Fn",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1850755 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1850869 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1850870 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1852320 T5908 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 1852321 T5908 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1852337 T5908 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1852340 T5908 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:51079
[junit4] 2> 1852346 T5908 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1852346 T5908 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1852347 T5908 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003
[junit4] 2> 1852347 T5908 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003/'
[junit4] 2> 1852403 T5908 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003/solr.xml
[junit4] 2> 1852428 T5908 oasc.CoreContainer.<init> New CoreContainer 1492392607
[junit4] 2> 1852429 T5908 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003/]
[junit4] 2> 1852430 T5908 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1852430 T5908 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1852431 T5908 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1852432 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1852432 T5908 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1852432 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1852432 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1852433 T5908 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1852434 T5908 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1852439 T5908 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1852440 T5908 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1852441 T5908 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1852441 T5908 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51071/solr
[junit4] 2> 1852442 T5908 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1852444 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1852454 T5952 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3298ee98 name:ZooKeeperConnection Watcher:127.0.0.1:51071 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1852454 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1852464 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1852468 T5954 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@188995ad name:ZooKeeperConnection Watcher:127.0.0.1:51071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1852469 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1852491 T5908 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1853500 T5908 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51079__v%2Fn
[junit4] 2> 1853502 T5908 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51079__v%2Fn
[junit4] 2> 1853535 T5955 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1853535 T5955 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1853540 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1853541 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1853542 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1853540 T5955 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1853551 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1853565 T5933 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:51079/_v/n",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51079__v%2Fn",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1853565 T5933 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 1853565 T5933 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1853573 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1853576 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1853577 T5954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1853577 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1854546 T5955 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1854547 T5955 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1854548 T5955 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1854548 T5955 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1854550 T5955 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1854550 T5955 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003/collection1/'
[junit4] 2> 1854551 T5955 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 1854552 T5955 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 1854552 T5955 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 1854670 T5955 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 1854695 T5955 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1854706 T5955 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1854720 T5955 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1854951 T5955 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1854952 T5955 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1854954 T5955 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1854959 T5955 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1855008 T5955 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1855008 T5955 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1855009 T5955 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty1/
[junit4] 2> 1855009 T5955 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@fcc0205
[junit4] 2> 1855010 T5955 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty1
[junit4] 2> 1855011 T5955 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty1/index/
[junit4] 2> 1855012 T5955 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 1855014 T5955 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty1/index
[junit4] 2> 1855015 T5955 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=67.3037109375, floorSegmentMB=1.296875, forceMergeDeletesPctAllowed=18.754281475137972, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1855019 T5955 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1855019 T5955 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1855025 T5955 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1855026 T5955 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1855026 T5955 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1855027 T5955 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1855027 T5955 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1855027 T5955 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1855029 T5955 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1855029 T5955 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1855030 T5955 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1855030 T5955 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1855030 T5955 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1855031 T5955 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1855031 T5955 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1855032 T5955 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1855033 T5955 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1855033 T5955 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1855034 T5955 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 1855042 T5955 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1855044 T5955 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1855044 T5955 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1855047 T5955 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=17, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7277011005546191]
[junit4] 2> 1855052 T5955 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1855053 T5955 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1855053 T5955 oass.SolrIndexSearcher.<init> Opening Searcher@4ad53d77[collection1] main
[junit4] 2> 1855054 T5955 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1855058 T5955 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1855058 T5955 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1855059 T5955 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1855059 T5955 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1855059 T5955 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1855061 T5955 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 1855061 T5955 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1855061 T5955 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 1855062 T5955 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1855070 T5956 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4ad53d77[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1855070 T5955 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1855072 T5959 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:51079/_v/n collection:collection1 shard:shard2
[junit4] 2> 1855074 T5908 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 1855074 T5908 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1855082 T5959 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1855133 T5959 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1855154 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1855159 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1855161 T5959 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1855162 T5959 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C3552 name=collection1 org.apache.solr.core.SolrCore@3f52c174 url=https://127.0.0.1:51079/_v/n/collection1 node=127.0.0.1:51079__v%2Fn C3552_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:51079/_v/n, core=collection1, node_name=127.0.0.1:51079__v%2Fn}
[junit4] 2> 1855162 T5959 C3552 P51079 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:51079/_v/n/collection1/
[junit4] 2> 1855162 T5959 C3552 P51079 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1855163 T5959 C3552 P51079 oasc.SyncStrategy.syncToMe https://127.0.0.1:51079/_v/n/collection1/ has no replicas
[junit4] 2> 1855163 T5959 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:51079/_v/n/collection1/ shard2
[junit4] 2> 1855163 T5959 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1855176 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1855178 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1855178 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1855181 T5954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1855208 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1855223 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1855234 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1855342 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1855342 T5954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1855343 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1855375 T5959 oasc.ZkController.register We are https://127.0.0.1:51079/_v/n/collection1/ and leader is https://127.0.0.1:51079/_v/n/collection1/
[junit4] 2> 1855376 T5959 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:51079/_v/n
[junit4] 2> 1855376 T5959 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1855376 T5959 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 1855376 T5959 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1855388 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1855392 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1855393 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1855396 T5959 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1855405 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1855409 T5933 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:51079/_v/n",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51079__v%2Fn",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1855428 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1855534 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1855535 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1855535 T5954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1856974 T5908 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 1856975 T5908 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1856982 T5908 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1856985 T5908 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:51082
[junit4] 2> 1856989 T5908 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1856989 T5908 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1856990 T5908 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004
[junit4] 2> 1856990 T5908 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004/'
[junit4] 2> 1857059 T5908 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004/solr.xml
[junit4] 2> 1857076 T5908 oasc.CoreContainer.<init> New CoreContainer 959363488
[junit4] 2> 1857076 T5908 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004/]
[junit4] 2> 1857077 T5908 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1857078 T5908 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1857078 T5908 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1857078 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1857079 T5908 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1857079 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1857079 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1857080 T5908 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1857080 T5908 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1857083 T5908 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1857084 T5908 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1857084 T5908 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1857085 T5908 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51071/solr
[junit4] 2> 1857086 T5908 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1857099 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1857100 T5970 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c03ef40 name:ZooKeeperConnection Watcher:127.0.0.1:51071 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1857101 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1857108 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1857111 T5972 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@728db7b5 name:ZooKeeperConnection Watcher:127.0.0.1:51071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1857112 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1857133 T5908 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1858145 T5908 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51082__v%2Fn
[junit4] 2> 1858155 T5908 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51082__v%2Fn
[junit4] 2> 1858177 T5973 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1858178 T5973 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1858179 T5973 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1858180 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1858180 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1858180 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1858184 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1858186 T5933 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:51082/_v/n",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51082__v%2Fn",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1858187 T5933 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1858187 T5933 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1858191 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1858194 T5972 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1858195 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1858197 T5954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1858198 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1859181 T5973 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1859182 T5973 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1859184 T5973 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1859184 T5973 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1859186 T5973 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1859187 T5973 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004/collection1/'
[junit4] 2> 1859188 T5973 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 1859188 T5973 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 1859189 T5973 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 1859238 T5973 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 1859262 T5973 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1859266 T5973 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1859284 T5973 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1859456 T5973 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1859456 T5973 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1859458 T5973 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1859466 T5973 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1859532 T5973 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1859532 T5973 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1859533 T5973 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty2/
[junit4] 2> 1859534 T5973 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@fcc0205
[junit4] 2> 1859535 T5973 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty2
[junit4] 2> 1859536 T5973 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty2/index/
[junit4] 2> 1859537 T5973 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 1859539 T5973 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty2/index
[junit4] 2> 1859539 T5973 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=67.3037109375, floorSegmentMB=1.296875, forceMergeDeletesPctAllowed=18.754281475137972, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1859544 T5973 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1859545 T5973 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1859550 T5973 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1859550 T5973 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1859551 T5973 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1859551 T5973 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1859552 T5973 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1859553 T5973 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1859554 T5973 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1859555 T5973 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1859555 T5973 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1859556 T5973 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1859556 T5973 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1859556 T5973 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1859557 T5973 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1859557 T5973 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1859557 T5973 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1859558 T5973 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1859559 T5973 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 1859570 T5973 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1859573 T5973 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1859574 T5973 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1859575 T5973 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=17, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7277011005546191]
[junit4] 2> 1859576 T5973 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1859576 T5973 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1859577 T5973 oass.SolrIndexSearcher.<init> Opening Searcher@725d9238[collection1] main
[junit4] 2> 1859577 T5973 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1859580 T5973 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1859580 T5973 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1859582 T5973 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1859582 T5973 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1859583 T5973 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1859586 T5973 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 1859586 T5973 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1859586 T5973 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 1859587 T5973 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1859591 T5974 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@725d9238[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1859594 T5973 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1859597 T5977 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:51082/_v/n collection:collection1 shard:shard3
[junit4] 2> 1859598 T5908 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 1859600 T5908 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1859602 T5977 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 1859617 T5977 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 1859635 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1859641 T5977 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1859642 T5977 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C3553 name=collection1 org.apache.solr.core.SolrCore@c8ec20e url=https://127.0.0.1:51082/_v/n/collection1 node=127.0.0.1:51082__v%2Fn C3553_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:51082/_v/n, core=collection1, node_name=127.0.0.1:51082__v%2Fn}
[junit4] 2> 1859643 T5977 C3553 P51082 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:51082/_v/n/collection1/
[junit4] 2> 1859645 T5977 C3553 P51082 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1859645 T5977 C3553 P51082 oasc.SyncStrategy.syncToMe https://127.0.0.1:51082/_v/n/collection1/ has no replicas
[junit4] 2> 1859646 T5977 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:51082/_v/n/collection1/ shard3
[junit4] 2> 1859646 T5977 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 1859650 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1859664 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1859675 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1859685 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1859791 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1859792 T5972 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1859792 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1859792 T5954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1859830 T5977 oasc.ZkController.register We are https://127.0.0.1:51082/_v/n/collection1/ and leader is https://127.0.0.1:51082/_v/n/collection1/
[junit4] 2> 1859830 T5977 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:51082/_v/n
[junit4] 2> 1859830 T5977 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1859831 T5977 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 1859831 T5977 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1859833 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1859833 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1859834 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1859834 T5977 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1859837 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1859839 T5933 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:51082/_v/n",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51082__v%2Fn",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 1859845 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1859951 T5954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1859952 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1859952 T5972 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1859952 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1861255 T5908 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 1861256 T5908 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1861264 T5908 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1861266 T5908 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:51085
[junit4] 2> 1861270 T5908 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1861271 T5908 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1861271 T5908 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-005
[junit4] 2> 1861271 T5908 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-005/'
[junit4] 2> 1861335 T5908 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-005/solr.xml
[junit4] 2> 1861364 T5908 oasc.CoreContainer.<init> New CoreContainer 2132999269
[junit4] 2> 1861365 T5908 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-005/]
[junit4] 2> 1861366 T5908 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1861367 T5908 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1861367 T5908 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1861367 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1861368 T5908 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1861369 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1861369 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1861369 T5908 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1861370 T5908 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1861376 T5908 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1861377 T5908 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1861377 T5908 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1861378 T5908 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51071/solr
[junit4] 2> 1861378 T5908 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1861381 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1861385 T5988 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54750aa7 name:ZooKeeperConnection Watcher:127.0.0.1:51071 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1861385 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1861393 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1861398 T5990 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@525f9ace name:ZooKeeperConnection Watcher:127.0.0.1:51071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1861399 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1861447 T5908 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1862455 T5908 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51085__v%2Fn
[junit4] 2> 1862465 T5908 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51085__v%2Fn
[junit4] 2> 1862544 T5991 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1862544 T5991 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1862548 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1862548 T5991 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1862548 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1862549 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1862553 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1862555 T5933 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:51085/_v/n",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51085__v%2Fn",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1862555 T5933 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1862555 T5933 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1862562 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1862564 T5972 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1862565 T5954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1862565 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1862565 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1862565 T5990 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1863550 T5991 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1863551 T5991 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1863552 T5991 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1863553 T5991 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1863556 T5991 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1863556 T5991 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafe
[...truncated too long message...]
=segments_6,generation=6}
[junit4] 2> 2019685 T6794 C4312 P51122 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4] 2> 2019699 T6794 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 2019700 T6794 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 2019701 T6794 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty10 [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty10;done=false>>]
[junit4] 2> 2019701 T6794 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty10
[junit4] 2> 2019701 T6794 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty10/index.20140504012554990 [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty10/index.20140504012554990;done=false>>]
[junit4] 2> 2019702 T6794 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty10/index.20140504012554990
[junit4] 2> 2019702 T6794 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 2019704 T6794 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001
[junit4] 2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=443), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=443)), sim=DefaultSimilarity, locale=nl_NL, timezone=ROK
[junit4] 2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=2,threads=1,free=65089976,total=317382656
[junit4] 2> NOTE: All tests run in this JVM: [TestRTGBase, SpellCheckCollatorTest, IndexSchemaTest, PathHierarchyTokenizerFactoryTest, AddSchemaFieldsUpdateProcessorFactoryTest, AtomicUpdatesTest, ShowFileRequestHandlerTest, TestReloadAndDeleteDocs, TestSweetSpotSimilarityFactory, SolrXmlInZkTest, TestOverriddenPrefixQueryForCustomFieldType, TestQuerySenderListener, TermVectorComponentDistributedTest, TestIndexSearcher, HdfsBasicDistributedZk2Test, AliasIntegrationTest, SuggestComponentTest, OpenCloseCoreStressTest, ShardSplitTest, AnalysisErrorHandlingTest, SimplePostToolTest, TestReload, CustomCollectionTest, BadIndexSchemaTest, TestUniqueKeyFieldResource, TestComplexPhraseQParserPlugin, PluginInfoTest, TestCloudInspectUtil, UnloadDistributedZkTest, TestAddFieldRealTimeGet, HdfsWriteToMultipleCollectionsTest, TestFoldingMultitermQuery, ReturnFieldsTest, SolrInfoMBeanTest, SliceStateUpdateTest, QueryParsingTest, BasicDistributedZkTest, TestCoreDiscovery, SearchHandlerTest, TestCustomSort, ClusterStateUpdateTest, TestWordDelimiterFilterFactory, SolrCoreCheckLockOnStartupTest, CachingDirectoryFactoryTest, TestRestManager, RequiredFieldsTest, DistribCursorPagingTest, SolrIndexSplitterTest, TestSolrQueryParserDefaultOperatorResource, ShardRoutingCustomTest, FieldMutatingUpdateProcessorTest, SOLR749Test, TestSerializedLuceneMatchVersion, TestDefaultSearchFieldResource, TestLMJelinekMercerSimilarityFactory, TestSolrDeletionPolicy2, DocumentAnalysisRequestHandlerTest, TestInitQParser, AbstractAnalyticsStatsTest, TestSchemaVersionResource, AddBlockUpdateTest, TestSurroundQueryParser, TestRangeQuery, TestManagedStopFilterFactory, CursorMarkTest, TestMiniSolrCloudCluster, TestJoin, DocValuesMultiTest, CursorPagingTest, TestRandomDVFaceting, SampleTest, RemoteQueryErrorTest, TestDFRSimilarityFactory, TestBM25SimilarityFactory, TestDistributedSearch, SchemaVersionSpecificBehaviorTest, ParsingFieldUpdateProcessorsTest, AnalysisAfterCoreReloadTest, TestSearcherReuse, BadComponentTest, TestFastOutputStream, TestSolrDeletionPolicy1, TestPartialUpdateDeduplication, TestHighlightDedupGrouping, TestLFUCache, ShardRoutingTest, TestGroupingSearch, NoFacetTest, TestTrie, TestCodecSupport, FullSolrCloudDistribCmdsTest, MinimalSchemaTest, HighlighterTest, TestShardHandlerFactory, BasicZkTest, TestPseudoReturnFields, HdfsBasicDistributedZkTest, TestBlendedInfixSuggestions, DistributedQueryElevationComponentTest, TestArbitraryIndexDir, TestSchemaSimilarityResource, InfoHandlerTest, TestFieldSortValues, FieldAnalysisRequestHandlerTest, BlockDirectoryTest, TestModifyConfFiles, DistribDocExpirationUpdateProcessorTest, IndexBasedSpellCheckerTest, HdfsChaosMonkeySafeLeaderTest, ResourceLoaderTest, XmlUpdateRequestHandlerTest, TestPerFieldSimilarity, ScriptEngineTest, ClusterStateTest, TestBadConfig, TestManagedSynonymFilterFactory, SimpleFacetsTest, SolrPluginUtilsTest, TestNoOpRegenerator, TimeZoneUtilsTest, TestXIncludeConfig, TestCloudManagedSchemaAddField, TestRequestStatusCollectionAPI, JsonLoaderTest, DistributedDebugComponentTest, TestDynamicFieldCollectionResource, BasicDistributedZk2Test, NoCacheHeaderTest, TestCollapseQParserPlugin, DistributedQueryComponentOptimizationTest, FastVectorHighlighterTest, RecoveryZkTest, TestSolrQueryParser, OpenExchangeRatesOrgProviderTest, SolrTestCaseJ4Test, NumericFieldsTest, TestManagedResourceStorage, SpellCheckComponentTest, ChangedSchemaMergeTest, CoreAdminHandlerTest, FunctionTest, TestRealTimeGet, SortByFunctionTest, TestSolrXMLSerializer, DistributedSuggestComponentTest, CSVRequestHandlerTest, DocValuesTest, TestRecoveryHdfs, TestSolrXmlPersistor, RegexBoostProcessorTest, TestFiltering, OutputWriterTest, TestPHPSerializedResponseWriter, SpellingQueryConverterTest, TriLevelCompositeIdRoutingTest, RequestHandlersTest, RangeFacetTest, RAMDirectoryFactoryTest, DirectUpdateHandlerOptimizeTest, TestSearchPerf, BinaryUpdateRequestHandlerTest, SpatialFilterTest, QueryEqualityTest, TestDistributedGrouping, EchoParamsTest, TestCursorMarkWithoutUniqueKey, TestQueryTypes, TestConfigSets, BasicFunctionalityTest, TestValueSourceCache, MoreLikeThisHandlerTest, CurrencyFieldOpenExchangeTest, TestLuceneMatchVersion, TestDocSet, CacheHeaderTest, DebugComponentTest, TestUtils, DeleteInactiveReplicaTest, HdfsLockFactoryTest, TestComponentsName, NotRequiredUniqueKeyTest, TestSolrXmlPersistence, FieldFacetTest, TestStressReorder, HdfsDirectoryTest, TestFieldCollectionResource, SuggesterTSTTest, AutoCommitTest, TestInfoStreamLogging, QueryResultKeyTest, TestIBSimilarityFactory, LoggingHandlerTest, ChaosMonkeySafeLeaderTest]
[junit4] Completed in 173.17s, 1 test, 1 failure <<< FAILURES!
[...truncated 618 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1296: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:920: There were test failures: 398 suites, 1670 tests, 1 failure, 42 ignored (15 assumptions)
Total time: 108 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure