You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/06/07 13:53:44 UTC
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_60) - Build #
10496 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10496/
Java: 32bit/jdk1.7.0_60 -client -XX:+UseParallelGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard3 is not consistent. Got 156 from http://127.0.0.1:51911/_npk/i/collection1lastClient and got 136 from http://127.0.0.1:36734/_npk/i/collection1
Stack Trace:
java.lang.AssertionError: shard3 is not consistent. Got 156 from http://127.0.0.1:51911/_npk/i/collection1lastClient and got 136 from http://127.0.0.1:36734/_npk/i/collection1
at __randomizedtesting.SeedInfo.seed([FE86C27786FEA910:7F604C6FF1A1C92C]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1131)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1110)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 10646 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/init-core-data-001
[junit4] 2> 10466 T62 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
[junit4] 2> 10466 T62 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_npk/i
[junit4] 2> 10472 T62 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 10475 T62 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 10477 T63 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 10576 T62 oasc.ZkTestServer.run start zk server on port:45576
[junit4] 2> 10613 T62 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 10701 T69 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11a32ad name:ZooKeeperConnection Watcher:127.0.0.1:45576 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 10701 T62 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 10703 T62 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 10727 T62 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 10729 T71 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f3a90 name:ZooKeeperConnection Watcher:127.0.0.1:45576/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 10729 T62 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 10732 T62 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 10736 T62 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 10739 T62 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 10741 T62 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 10744 T62 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 10752 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 10757 T62 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 10758 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 10764 T62 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 10764 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 10769 T62 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 10770 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 10773 T62 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 10774 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 10777 T62 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 10777 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 10781 T62 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 10782 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 10785 T62 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 10785 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 10788 T62 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 10788 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 10792 T62 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 10792 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 10795 T62 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 10795 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 11081 T62 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 11084 T62 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37828
[junit4] 2> 11084 T62 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 11084 T62 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 11085 T62 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-002
[junit4] 2> 11085 T62 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-002/'
[junit4] 2> 11097 T62 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-002/solr.xml
[junit4] 2> 11125 T62 oasc.CoreContainer.<init> New CoreContainer 18345888
[junit4] 2> 11126 T62 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-002/]
[junit4] 2> 11127 T62 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 11127 T62 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 11127 T62 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 11128 T62 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 11128 T62 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 11128 T62 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 11129 T62 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 11129 T62 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 11129 T62 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 11130 T62 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 11130 T62 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 11131 T62 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 11131 T62 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45576/solr
[junit4] 2> 11140 T62 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 11141 T62 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 11143 T82 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@127f54a name:ZooKeeperConnection Watcher:127.0.0.1:45576 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 11143 T62 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 11148 T62 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 11150 T84 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19a6ec9 name:ZooKeeperConnection Watcher:127.0.0.1:45576/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 11150 T62 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 11161 T62 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 11170 T62 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 11178 T62 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 11184 T62 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 11190 T62 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 11199 T62 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 11201 T62 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37828__npk%2Fi
[junit4] 2> 11204 T62 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37828__npk%2Fi
[junit4] 2> 11209 T62 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 11212 T62 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 11221 T62 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:37828__npk%2Fi
[junit4] 2> 11221 T62 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 11224 T62 oasc.Overseer.start Overseer (id=91890666082926595-127.0.0.1:37828__npk%2Fi-n_0000000000) starting
[junit4] 2> 11232 T62 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 11252 T86 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 11253 T62 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 11255 T62 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 11258 T62 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 11263 T85 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 11268 T87 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 11269 T87 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 11272 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 11272 T87 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 11274 T85 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 11275 T85 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:37828/_npk/i",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:37828__npk%2Fi",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 11276 T85 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 11277 T85 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 11281 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 11288 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 12272 T87 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 12273 T87 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 12274 T87 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 12274 T87 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 12277 T87 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 12277 T87 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-002/collection1/'
[junit4] 2> 12278 T87 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 12278 T87 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 12279 T87 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 12353 T87 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 12382 T87 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 12385 T87 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 12392 T87 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 12617 T87 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 12619 T87 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 12620 T87 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 12623 T87 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 12650 T87 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 12650 T87 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 12662 T87 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/control/data/
[junit4] 2> 12663 T87 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@18eaab5
[junit4] 2> 12688 T87 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/control/data
[junit4] 2> 12688 T87 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/control/data/index/
[junit4] 2> 12689 T87 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 12689 T87 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/control/data/index
[junit4] 2> 12690 T87 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=2129121542, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 12695 T87 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 12695 T87 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 12703 T87 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 12716 T87 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 12716 T87 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 12717 T87 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 12717 T87 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 12717 T87 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 12718 T87 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 12718 T87 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 12718 T87 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 12719 T87 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 12720 T87 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 12720 T87 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 12721 T87 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 12721 T87 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 12722 T87 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 12723 T87 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 12723 T87 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 12726 T87 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 12728 T87 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 12728 T87 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 12730 T87 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=72.8056640625, floorSegmentMB=0.353515625, forceMergeDeletesPctAllowed=6.246513242740316, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 12731 T87 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 12731 T87 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 12731 T87 oass.SolrIndexSearcher.<init> Opening Searcher@156778c[collection1] main
[junit4] 2> 12731 T87 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 12733 T87 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 12733 T87 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 12735 T87 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 12736 T87 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 12736 T87 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 12737 T87 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 12737 T87 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 12737 T87 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 12738 T87 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 12743 T88 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@156778c[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 12748 T87 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 12749 T91 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37828/_npk/i collection:control_collection shard:shard1
[junit4] 2> 12749 T62 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 12750 T62 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 12755 T91 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 12762 T91 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 12763 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 12764 T91 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 12764 T62 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 12765 T91 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 12765 T85 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C5 name=collection1 org.apache.solr.core.SolrCore@19021c5 url=http://127.0.0.1:37828/_npk/i/collection1 node=127.0.0.1:37828__npk%2Fi C5_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:37828/_npk/i, core=collection1, node_name=127.0.0.1:37828__npk%2Fi}
[junit4] 2> 12765 T91 C5 P37828 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:37828/_npk/i/collection1/
[junit4] 2> 12766 T91 C5 P37828 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 12765 T93 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cebc44 name:ZooKeeperConnection Watcher:127.0.0.1:45576/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 12766 T91 C5 P37828 oasc.SyncStrategy.syncToMe http://127.0.0.1:37828/_npk/i/collection1/ has no replicas
[junit4] 2> 12767 T62 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 12767 T91 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:37828/_npk/i/collection1/ shard1
[junit4] 2> 12770 T91 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 12771 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 12771 T62 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 12776 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 12777 T62 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 12780 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 12882 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 12882 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 12928 T91 oasc.ZkController.register We are http://127.0.0.1:37828/_npk/i/collection1/ and leader is http://127.0.0.1:37828/_npk/i/collection1/
[junit4] 2> 12928 T91 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37828/_npk/i
[junit4] 2> 12928 T91 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 12929 T91 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 12929 T91 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 12931 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 12931 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 12931 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 12931 T91 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 12933 T85 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 12934 T85 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:37828/_npk/i",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:37828__npk%2Fi",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 12936 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 13043 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 13043 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 13063 T62 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 13063 T62 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 13068 T62 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34818
[junit4] 2> 13068 T62 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 13069 T62 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 13069 T62 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-003
[junit4] 2> 13069 T62 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-003/'
[junit4] 2> 13085 T62 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-003/solr.xml
[junit4] 2> 13115 T62 oasc.CoreContainer.<init> New CoreContainer 19595181
[junit4] 2> 13115 T62 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-003/]
[junit4] 2> 13116 T62 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 13116 T62 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 13117 T62 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 13117 T62 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 13117 T62 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 13117 T62 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 13118 T62 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 13118 T62 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 13118 T62 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 13119 T62 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 13119 T62 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 13120 T62 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 13120 T62 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45576/solr
[junit4] 2> 13120 T62 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 13121 T62 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 13123 T104 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f7bb8e name:ZooKeeperConnection Watcher:127.0.0.1:45576 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 13123 T62 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 13125 T62 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 13126 T106 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f542e9 name:ZooKeeperConnection Watcher:127.0.0.1:45576/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 13127 T62 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 13137 T62 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 14141 T62 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34818__npk%2Fi
[junit4] 2> 14143 T62 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34818__npk%2Fi
[junit4] 2> 14163 T107 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 14163 T107 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 14165 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14165 T107 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 14166 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14167 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14170 T85 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 14171 T85 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:34818/_npk/i",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:34818__npk%2Fi",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 14172 T85 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 14172 T85 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 14175 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14277 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 14277 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 14277 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 15166 T107 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 15167 T107 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 15167 T107 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 15168 T107 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 15169 T107 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 15169 T107 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-003/collection1/'
[junit4] 2> 15170 T107 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 15170 T107 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 15170 T107 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 15204 T107 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 15232 T107 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 15234 T107 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 15241 T107 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 15460 T107 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 15462 T107 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 15463 T107 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 15467 T107 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 15488 T107 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 15488 T107 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 15493 T107 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty1/
[junit4] 2> 15493 T107 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@18eaab5
[junit4] 2> 15494 T107 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty1
[junit4] 2> 15495 T107 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty1/index/
[junit4] 2> 15495 T107 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 15495 T107 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty1/index
[junit4] 2> 15496 T107 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=2129121542, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 15506 T107 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 15506 T107 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 15515 T107 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 15515 T107 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 15516 T107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 15516 T107 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 15516 T107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 15516 T107 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 15517 T107 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 15517 T107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 15517 T107 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 15518 T107 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 15519 T107 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 15519 T107 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 15520 T107 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 15521 T107 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 15521 T107 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 15522 T107 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 15524 T107 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 15526 T107 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 15528 T107 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 15528 T107 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 15530 T107 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=72.8056640625, floorSegmentMB=0.353515625, forceMergeDeletesPctAllowed=6.246513242740316, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 15531 T107 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 15532 T107 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 15532 T107 oass.SolrIndexSearcher.<init> Opening Searcher@32abfc[collection1] main
[junit4] 2> 15532 T107 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 15533 T107 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 15534 T107 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 15534 T107 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 15534 T107 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 15535 T107 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 15535 T107 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 15536 T107 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 15536 T107 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 15536 T107 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 15542 T108 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@32abfc[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 15546 T107 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 15547 T111 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:34818/_npk/i collection:collection1 shard:shard2
[junit4] 2> 15548 T62 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 15548 T111 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 15548 T62 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 15554 T111 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 15556 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15556 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15556 T111 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 15557 T111 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 15557 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15557 T85 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C6 name=collection1 org.apache.solr.core.SolrCore@16027fa url=http://127.0.0.1:34818/_npk/i/collection1 node=127.0.0.1:34818__npk%2Fi C6_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:34818/_npk/i, core=collection1, node_name=127.0.0.1:34818__npk%2Fi}
[junit4] 2> 15557 T111 C6 P34818 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:34818/_npk/i/collection1/
[junit4] 2> 15558 T111 C6 P34818 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 15558 T111 C6 P34818 oasc.SyncStrategy.syncToMe http://127.0.0.1:34818/_npk/i/collection1/ has no replicas
[junit4] 2> 15559 T111 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:34818/_npk/i/collection1/ shard2
[junit4] 2> 15559 T111 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 15560 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15563 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15566 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15669 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 15669 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 15669 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 15715 T111 oasc.ZkController.register We are http://127.0.0.1:34818/_npk/i/collection1/ and leader is http://127.0.0.1:34818/_npk/i/collection1/
[junit4] 2> 15715 T111 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:34818/_npk/i
[junit4] 2> 15715 T111 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 15715 T111 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 15716 T111 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 15717 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15718 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15718 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15718 T111 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 15719 T85 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 15720 T85 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:34818/_npk/i",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:34818__npk%2Fi",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 15722 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15802 T62 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 15803 T62 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 15805 T62 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41582
[junit4] 2> 15806 T62 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 15806 T62 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 15806 T62 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-004
[junit4] 2> 15807 T62 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-004/'
[junit4] 2> 15821 T62 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-004/solr.xml
[junit4] 2> 15828 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 15828 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 15828 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 15850 T62 oasc.CoreContainer.<init> New CoreContainer 16894856
[junit4] 2> 15851 T62 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-004/]
[junit4] 2> 15852 T62 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 15852 T62 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 15852 T62 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 15852 T62 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 15853 T62 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 15853 T62 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 15853 T62 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 15854 T62 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 15854 T62 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 15855 T62 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 15855 T62 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 15855 T62 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 15856 T62 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45576/solr
[junit4] 2> 15856 T62 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 15857 T62 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 15858 T122 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d9556a name:ZooKeeperConnection Watcher:127.0.0.1:45576 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 15858 T62 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 15861 T62 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 15862 T124 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b4467 name:ZooKeeperConnection Watcher:127.0.0.1:45576/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 15862 T62 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 15867 T62 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 16870 T62 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41582__npk%2Fi
[junit4] 2> 16872 T62 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41582__npk%2Fi
[junit4] 2> 16891 T125 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 16892 T125 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 16894 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 16894 T125 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 16894 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 16895 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 16896 T85 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 16898 T85 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:41582/_npk/i",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:41582__npk%2Fi",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 16898 T85 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 16899 T85 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 16904 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 17008 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 17008 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 17008 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 17008 T124 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 17894 T125 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 17895 T125 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 17896 T125 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 17896 T125 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 17897 T125 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 17897 T125 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-004/collection1/'
[junit4] 2> 17898 T125 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 17898 T125 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 17898 T125 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 17933 T125 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 17961 T125 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 17963 T125 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 17970 T125 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 18188 T125 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 18190 T125 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 18192 T125 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 18195 T125 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 18219 T125 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 18220 T125 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 18220 T125 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty2/
[junit4] 2> 18221 T125 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@18eaab5
[junit4] 2> 18222 T125 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty2
[junit4] 2> 18223 T125 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty2/index/
[junit4] 2> 18223 T125 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 18224 T125 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty2/index
[junit4] 2> 18225 T125 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=2129121542, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 18235 T125 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 18235 T125 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 18245 T125 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 18246 T125 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 18246 T125 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 18247 T125 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 18247 T125 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 18248 T125 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 18248 T125 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 18249 T125 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 18249 T125 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 18251 T125 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 18252 T125 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 18252 T125 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 18252 T125 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 18253 T125 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 18253 T125 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 18254 T125 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 18255 T125 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 18257 T125 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 18259 T125 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 18259 T125 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 18261 T125 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=72.8056640625, floorSegmentMB=0.353515625, forceMergeDeletesPctAllowed=6.246513242740316, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 18262 T125 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 18262 T125 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 18262 T125 oass.SolrIndexSearcher.<init> Opening Searcher@51fb0b[collection1] main
[junit4] 2> 18262 T125 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 18264 T125 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 18264 T125 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 18265 T125 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 18265 T125 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 18266 T125 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 18266 T125 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 18267 T125 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 18267 T125 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 18267 T125 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 18272 T126 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@51fb0b[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 18277 T125 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 18278 T129 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41582/_npk/i collection:collection1 shard:shard3
[junit4] 2> 18279 T62 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 18279 T62 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 18279 T129 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 18284 T129 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 18286 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 18286 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 18286 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 18287 T129 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 18287 T129 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C7 name=collection1 org.apache.solr.core.SolrCore@195f05 url=http://127.0.0.1:41582/_npk/i/collection1 node=127.0.0.1:41582__npk%2Fi C7_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:41582/_npk/i, core=collection1, node_name=127.0.0.1:41582__npk%2Fi}
[junit4] 2> 18287 T129 C7 P41582 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:41582/_npk/i/collection1/
[junit4] 2> 18287 T129 C7 P41582 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 18287 T85 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 18288 T129 C7 P41582 oasc.SyncStrategy.syncToMe http://127.0.0.1:41582/_npk/i/collection1/ has no replicas
[junit4] 2> 18288 T129 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:41582/_npk/i/collection1/ shard3
[junit4] 2> 18288 T129 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 18290 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 18292 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 18294 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 18397 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 18397 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 18397 T124 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 18397 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 18443 T129 oasc.ZkController.register We are http://127.0.0.1:41582/_npk/i/collection1/ and leader is http://127.0.0.1:41582/_npk/i/collection1/
[junit4] 2> 18443 T129 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41582/_npk/i
[junit4] 2> 18443 T129 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 18444 T129 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 18444 T129 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 18447 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 18447 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 18448 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 18448 T129 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 18449 T85 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 18450 T85 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:41582/_npk/i",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:41582__npk%2Fi",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 18452 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 18535 T62 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 18536 T62 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 18538 T62 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44025
[junit4] 2> 18539 T62 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 18539 T62 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 18539 T62 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-005
[junit4] 2> 18540 T62 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-005/'
[junit4] 2> 18554 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 18554 T124 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 18554 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 18555 T62 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-005/solr.xml
[junit4] 2> 18554 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 18602 T62 oasc.CoreContainer.<init> New CoreContainer 6368299
[junit4] 2> 18602 T62 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-005/]
[junit4] 2> 18603 T62 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 18603 T62 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 18604 T62 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 18604 T62 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 18604 T62 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 18605 T62 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 18605 T62 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 18605 T62 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 18606 T62 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 18606 T62 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 18607 T62 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 18607 T62 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 18608 T62 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45576/solr
[junit4] 2> 18608 T62 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 18609 T62 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 18611 T140 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19af9d2 name:ZooKeeperConnection Watcher:127.0.0.1:45576 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 18612 T62 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 18616 T62 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 18618 T142 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f0e48e name:ZooKeeperConnection Watcher:127.0.0.1:45576/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 18618 T62 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 18631 T62 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 19634 T62 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44025__npk%2Fi
[junit4] 2> 19636 T62 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44025__npk%2Fi
[junit4] 2> 19646 T143 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 19646 T143 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 19647 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 19647 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 19647 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 19648 T143 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 19649 T85 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 19649 T85 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:44025/_npk/i",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:44025__npk%2Fi",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 19650 T85 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 19650 T85 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 19652 T84 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 19755 T142 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 19755 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 19755 T106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 19755 T124 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 19755 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 20648 T143 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 20648 T143 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 20649 T143 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 20650 T143 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[...truncated too long message...]
d5a url=http://127.0.0.1:36734/_npk/i/collection1 node=127.0.0.1:36734__npk%2Fi C387_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:36734/_npk/i, core=collection1, node_name=127.0.0.1:36734__npk%2Fi}
[junit4] 2> 86538 T576 C387 P36734 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
[junit4] 2> 86539 T576 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
[junit4] 2> 86539 T576 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@d52d5a
[junit4] 2> 86540 T576 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=59,cumulative_deletesById=22,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=14722,transaction_logs_total_number=3}
[junit4] 2> 86541 T576 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 86541 T576 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 86541 T576 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 86546 T576 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 86547 T576 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 86547 T576 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty11 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty11;done=false>>]
[junit4] 2> 86547 T576 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty11
[junit4] 2> 86548 T576 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty11/index.20140608012526133 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty11/index.20140608012526133;done=false>>]
[junit4] 2> 86548 T576 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty11/index.20140608012526133
[junit4] 2> 86548 T576 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 86548 T576 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> ASYNC NEW_CORE C388 name=collection1 org.apache.solr.core.SolrCore@16dce33 url=http://127.0.0.1:55909/_npk/i/collection1 node=127.0.0.1:55909__npk%2Fi C388_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:55909/_npk/i, core=collection1, node_name=127.0.0.1:55909__npk%2Fi}
[junit4] 2> 86786 T372 C388 P55909 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
[junit4] 2> 86787 T372 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
[junit4] 2> 86787 T372 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@16dce33
[junit4] 2> 86789 T372 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=290,cumulative_deletesById=140,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=24307,transaction_logs_total_number=1}
[junit4] 2> 86789 T372 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 86790 T372 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 86790 T372 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 86798 T372 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 86799 T372 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 86799 T372 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty10/index.20140608012532229 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty10/index.20140608012532229;done=false>>]
[junit4] 2> 86799 T372 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty10/index.20140608012532229
[junit4] 2> 86799 T372 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty10 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty10;done=false>>]
[junit4] 2> 86800 T372 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001/tempDir-001/jetty10
[junit4] 2> 86800 T372 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-FE86C27786FEA910-001
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=fi, timezone=Pacific/Kiritimati
[junit4] 2> NOTE: Linux 3.8.0-41-generic i386/Oracle Corporation 1.7.0_60 (32-bit)/cpus=8,threads=1,free=145596688,total=223346688
[junit4] 2> NOTE: All tests run in this JVM: [TestFieldTypeCollectionResource, DateMathParserTest, TestSolrDeletionPolicy2, HdfsBasicDistributedZkTest, TestNoOpRegenerator, TestComponentsName, InfoHandlerTest, AssignTest, HdfsRecoveryZkTest, TestSweetSpotSimilarityFactory, TestBinaryResponseWriter, ChaosMonkeySafeLeaderTest]
[junit4] Completed on J1 in 77.13s, 1 test, 1 failure <<< FAILURES!
[...truncated 1180 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1296: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:920: There were test failures: 410 suites, 1693 tests, 1 failure, 36 ignored (15 assumptions)
Total time: 66 minutes 13 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_05) - Build #
10497 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10497/
Java: 64bit/jdk1.8.0_05 -XX:+UseCompressedOops -XX:+UseG1GC
3 tests failed.
REGRESSION: org.apache.solr.cloud.OverseerTest.testShardAssignment
Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
at __randomizedtesting.SeedInfo.seed([C7D26DC779D1943:7565298C08C69D1F]:0)
at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:135)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
at org.apache.solr.cloud.OverseerTest.testShardAssignment(OverseerTest.java:232)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:127)
at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:122)
... 45 more
REGRESSION: org.apache.solr.cloud.UnloadDistributedZkTest.testDistribSearch
Error Message:
Leader never changed
Stack Trace:
java.lang.AssertionError: Leader never changed
at __randomizedtesting.SeedInfo.seed([C7D26DC779D1943:8D9BA8C400C2797F]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.UnloadDistributedZkTest.testCoreUnloadAndLeaders(UnloadDistributedZkTest.java:335)
at org.apache.solr.cloud.UnloadDistributedZkTest.doTest(UnloadDistributedZkTest.java:75)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 485 from https://127.0.0.1:48999/collection1lastClient and got 481 from https://127.0.0.1:41855/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 485 from https://127.0.0.1:48999/collection1lastClient and got 481 from https://127.0.0.1:41855/collection1
at __randomizedtesting.SeedInfo.seed([C7D26DC779D1943:8D9BA8C400C2797F]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1131)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1110)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 10647 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/init-core-data-001
[junit4] 2> 5897 T18 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 5908 T18 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 5950 T18 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 5955 T18 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 5978 T19 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 6176 T18 oasc.ZkTestServer.run start zk server on port:43149
[junit4] 2> 6331 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 6367 T20 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
[junit4] 2> 7916 T25 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@710a0aa9 name:ZooKeeperConnection Watcher:127.0.0.1:43149 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 7917 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 7918 T18 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 7946 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 7950 T27 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65df1232 name:ZooKeeperConnection Watcher:127.0.0.1:43149/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 7951 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 7956 T18 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 7963 T18 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 7968 T18 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 7972 T18 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 7977 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 7987 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 7994 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 7995 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 7999 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 8000 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 8009 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 8010 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 8014 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 8015 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 8019 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 8020 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 8026 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 8027 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 8032 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 8033 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 8037 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 8037 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 8044 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 8044 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 8050 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 8051 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 8059 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 8061 T29 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e52bf00 name:ZooKeeperConnection Watcher:127.0.0.1:43149/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 8062 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 8538 T18 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 8942 T18 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 8981 T18 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:58659
[junit4] 2> 9033 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 9034 T18 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 9034 T18 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-002
[junit4] 2> 9035 T18 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-002/'
[junit4] 2> 9055 T18 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-002/solr.xml
[junit4] 2> 9096 T18 oasc.CoreContainer.<init> New CoreContainer 426670685
[junit4] 2> 9097 T18 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-002/]
[junit4] 2> 9162 T18 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 9162 T18 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 9163 T18 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 9163 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 9164 T18 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 9165 T18 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 9165 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 9166 T18 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 9166 T18 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 9171 T18 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 9171 T18 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 9172 T18 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 9173 T18 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43149/solr
[junit4] 2> 9187 T18 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 9188 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 9194 T41 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56bce30 name:ZooKeeperConnection Watcher:127.0.0.1:43149 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 9194 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 9206 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 9208 T43 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6749076f name:ZooKeeperConnection Watcher:127.0.0.1:43149/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 9208 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 9221 T18 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 9231 T18 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 9239 T18 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 9250 T18 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 9254 T18 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 9273 T18 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 9277 T18 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58659_
[junit4] 2> 9289 T18 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58659_
[junit4] 2> 9294 T18 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 9300 T18 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 9313 T18 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:58659_
[junit4] 2> 9314 T18 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 9318 T18 oasc.Overseer.start Overseer (id=91891579775025156-127.0.0.1:58659_-n_0000000000) starting
[junit4] 2> 9332 T18 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 9359 T45 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 9361 T18 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 9367 T18 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 9369 T18 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 9382 T44 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 9389 T46 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 9389 T46 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 9395 T46 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 9399 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 9403 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 9408 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:58659",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:58659_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 9409 T44 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 9421 T44 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 9426 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 9435 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 10396 T46 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 10396 T46 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 10397 T46 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 10398 T46 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 10399 T46 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 10400 T46 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-002/collection1/'
[junit4] 2> 10401 T46 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 10402 T46 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 10402 T46 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 10467 T46 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 10487 T46 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 10489 T46 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 10504 T46 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 10761 T46 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 10763 T46 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 10765 T46 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 10770 T46 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 10810 T46 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 10811 T46 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 10818 T46 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/control/data/
[junit4] 2> 10821 T46 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3dea18ef
[junit4] 2> 10826 T46 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/control/data
[junit4] 2> 10827 T46 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/control/data/index/
[junit4] 2> 10827 T46 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 10828 T46 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/control/data/index
[junit4] 2> 10829 T46 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=63.80859375, floorSegmentMB=1.708984375, forceMergeDeletesPctAllowed=16.67279946973475, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 10834 T46 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 10834 T46 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 10844 T46 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 10857 T46 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 10857 T46 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 10857 T46 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 10858 T46 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 10858 T46 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 10859 T46 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 10859 T46 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 10859 T46 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 10860 T46 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 10862 T46 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 10862 T46 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 10862 T46 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 10863 T46 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 10864 T46 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 10869 T46 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 10877 T46 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 10879 T46 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 10891 T46 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 10892 T46 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 10895 T46 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=19, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 10897 T46 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 10897 T46 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 10898 T46 oass.SolrIndexSearcher.<init> Opening Searcher@6e6bf735[collection1] main
[junit4] 2> 10899 T46 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 10900 T46 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 10901 T46 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 10905 T46 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 10905 T46 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 10906 T46 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 10907 T46 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 10908 T46 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 10911 T46 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 10912 T46 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 10924 T47 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6e6bf735[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 10933 T46 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 10934 T50 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:58659 collection:control_collection shard:shard1
[junit4] 2> 10935 T18 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 10936 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 10943 T50 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 10952 T50 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 10956 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 10956 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 10958 T52 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d9207fd name:ZooKeeperConnection Watcher:127.0.0.1:43149/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 10958 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 10959 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 10960 T50 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 10962 T18 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 10965 T50 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@28e3f5b0 url=https://127.0.0.1:58659/collection1 node=127.0.0.1:58659_ C3_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:58659, core=collection1, node_name=127.0.0.1:58659_}
[junit4] 2> 10966 T50 C3 P58659 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:58659/collection1/
[junit4] 2> 10967 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 10969 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 10970 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 10972 T50 C3 P58659 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 10973 T50 C3 P58659 oasc.SyncStrategy.syncToMe https://127.0.0.1:58659/collection1/ has no replicas
[junit4] 2> 10973 T50 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:58659/collection1/ shard1
[junit4] 2> 10976 T50 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 10984 T18 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 10988 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 10990 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 10995 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 11100 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 11101 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 11143 T50 oasc.ZkController.register We are https://127.0.0.1:58659/collection1/ and leader is https://127.0.0.1:58659/collection1/
[junit4] 2> 11145 T50 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:58659
[junit4] 2> 11145 T50 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 11145 T50 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 11146 T50 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 11149 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 11150 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 11150 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 11153 T50 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 11155 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 11156 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:58659",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:58659_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 11162 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 11267 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 11268 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 11457 T18 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 11458 T18 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 11464 T18 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 11468 T18 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:36468
[junit4] 2> 11470 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 11471 T18 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 11471 T18 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-003
[junit4] 2> 11471 T18 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-003/'
[junit4] 2> 11486 T18 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-003/solr.xml
[junit4] 2> 11516 T18 oasc.CoreContainer.<init> New CoreContainer 132346164
[junit4] 2> 11517 T18 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-003/]
[junit4] 2> 11518 T18 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 11518 T18 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 11519 T18 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 11519 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 11519 T18 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 11520 T18 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 11520 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 11521 T18 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 11521 T18 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 11531 T18 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 11531 T18 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 11532 T18 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 11533 T18 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43149/solr
[junit4] 2> 11533 T18 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 11544 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 11546 T63 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37d4bfef name:ZooKeeperConnection Watcher:127.0.0.1:43149 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 11547 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 11559 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 11563 T65 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e77e338 name:ZooKeeperConnection Watcher:127.0.0.1:43149/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 11564 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 11580 T18 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 12585 T18 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36468_
[junit4] 2> 12586 T18 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36468_
[junit4] 2> 12596 T66 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 12596 T66 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 12597 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 12597 T66 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 12597 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 12598 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 12599 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 12600 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:36468",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:36468_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 12600 T44 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 12601 T44 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 12604 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 12707 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 12707 T65 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 12707 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 13598 T66 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 13598 T66 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 13599 T66 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 13599 T66 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 13601 T66 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 13601 T66 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-003/collection1/'
[junit4] 2> 13601 T66 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 13602 T66 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 13602 T66 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 13633 T66 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 13648 T66 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 13650 T66 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 13670 T66 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 13807 T66 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 13809 T66 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 13810 T66 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 13814 T66 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 13845 T66 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 13845 T66 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 13846 T66 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/jetty1/
[junit4] 2> 13846 T66 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3dea18ef
[junit4] 2> 13848 T66 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/jetty1
[junit4] 2> 13848 T66 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/jetty1/index/
[junit4] 2> 13848 T66 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 13849 T66 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/jetty1/index
[junit4] 2> 13850 T66 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=63.80859375, floorSegmentMB=1.708984375, forceMergeDeletesPctAllowed=16.67279946973475, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 13855 T66 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 13855 T66 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 13867 T66 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 13867 T66 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 13867 T66 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 13867 T66 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 13868 T66 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 13868 T66 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 13869 T66 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 13869 T66 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 13870 T66 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 13871 T66 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 13872 T66 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 13872 T66 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 13872 T66 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 13873 T66 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 13874 T66 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 13874 T66 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 13875 T66 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 13877 T66 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 13879 T66 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 13879 T66 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 13881 T66 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=19, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 13883 T66 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 13883 T66 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 13883 T66 oass.SolrIndexSearcher.<init> Opening Searcher@79b5fc66[collection1] main
[junit4] 2> 13884 T66 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 13885 T66 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 13885 T66 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 13886 T66 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 13886 T66 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 13887 T66 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 13887 T66 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 13888 T66 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 13888 T66 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 13888 T66 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 13895 T67 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79b5fc66[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 13902 T66 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 13903 T70 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:36468 collection:collection1 shard:shard2
[junit4] 2> 13904 T18 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 13904 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 13908 T70 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 13915 T70 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 13919 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 13919 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 13920 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 13920 T70 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 13921 T70 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 13921 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C4 name=collection1 org.apache.solr.core.SolrCore@31e037c4 url=https://127.0.0.1:36468/collection1 node=127.0.0.1:36468_ C4_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:36468, core=collection1, node_name=127.0.0.1:36468_}
[junit4] 2> 13921 T70 C4 P36468 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:36468/collection1/
[junit4] 2> 13923 T70 C4 P36468 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 13923 T70 C4 P36468 oasc.SyncStrategy.syncToMe https://127.0.0.1:36468/collection1/ has no replicas
[junit4] 2> 13923 T70 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:36468/collection1/ shard2
[junit4] 2> 13924 T70 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 13926 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 13942 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 13947 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14049 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 14050 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 14050 T65 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 14095 T70 oasc.ZkController.register We are https://127.0.0.1:36468/collection1/ and leader is https://127.0.0.1:36468/collection1/
[junit4] 2> 14095 T70 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:36468
[junit4] 2> 14095 T70 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 14095 T70 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 14096 T70 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 14102 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14102 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14103 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14103 T70 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 14104 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 14106 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:36468",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:36468_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 14111 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14214 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 14214 T65 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 14214 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 14286 T18 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 14287 T18 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 14295 T18 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 14300 T18 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:51590
[junit4] 2> 14305 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 14306 T18 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 14306 T18 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-004
[junit4] 2> 14307 T18 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-004/'
[junit4] 2> 14324 T18 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-004/solr.xml
[junit4] 2> 14379 T18 oasc.CoreContainer.<init> New CoreContainer 1972051392
[junit4] 2> 14379 T18 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-004/]
[junit4] 2> 14380 T18 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 14381 T18 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 14381 T18 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 14382 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 14382 T18 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 14382 T18 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 14383 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 14383 T18 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 14384 T18 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 14395 T18 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 14396 T18 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 14398 T18 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 14399 T18 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43149/solr
[junit4] 2> 14399 T18 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 14400 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 14403 T81 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d727982 name:ZooKeeperConnection Watcher:127.0.0.1:43149 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 14404 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 14418 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 14422 T83 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61251c08 name:ZooKeeperConnection Watcher:127.0.0.1:43149/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 14422 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 14431 T18 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 15437 T18 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51590_
[junit4] 2> 15439 T18 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51590_
[junit4] 2> 15454 T84 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 15454 T84 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 15457 T84 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 15457 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15457 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15458 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15460 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 15462 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:51590",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51590_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 15462 T44 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 15462 T44 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 15466 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15571 T83 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 15571 T52 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 15571 T65 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 15571 T43 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 16457 T84 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 16457 T84 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 16458 T84 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 16459 T84 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 16460 T84 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 16460 T84 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-004/collection1/'
[junit4] 2> 16461 T84 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 16461 T84 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 16462 T84 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 16491 T84 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 16506 T84 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 16507 T84 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 16527 T84 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 16668 T84 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 16670 T84 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 16671 T84 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 16676 T84 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 16710 T84 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 16711 T84 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 16711 T84 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/jetty2/
[junit4] 2> 16712 T84 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3dea18ef
[junit4] 2> 16713 T84 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/jetty2
[junit4] 2> 16714 T84 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/jetty2/index/
[junit4] 2> 16714 T84 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 16715 T84 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/jetty2/index
[junit4] 2> 16715 T84 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=63.80859375, floorSegmentMB=1.708984375, forceMergeDeletesPctAllowed=16.67279946973475, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 16720 T84 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 16721 T84 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 16731 T84 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 16731 T84 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 16732 T84 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 16732 T84 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 16732 T84 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 16732 T84 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 16736 T84 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 16737 T84 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 16737 T84 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 16739 T84 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 16740 T84 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 16740 T84 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 16740 T84 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 16741 T84 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 16742 T84 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 16742 T84 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 16743 T84 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 16745 T84 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 16746 T84 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 16746 T84 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 16748 T84 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=19, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 16749 T84 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C7D26DC779D1943-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 16749 T84 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 16749 T84 oass.SolrIndexSearcher.<init> Opening Searcher@291411d8[collection1] main
[junit4] 2> 16750 T84 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 16751 T84 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 16751 T84 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 16752 T84 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 16752 T84 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 16752 T84 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 16753 T84 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 16753 T84 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 16754 T84 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 16754 T84 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 16760 T85 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@291411d8[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 16766 T84 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 16767 T88 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:51590 collection:collection1 shard:shard3
[junit4] 2> 16767 T18 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 16768 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 16773 T88 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 16779 T88 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 16781 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 16782 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 16782 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 16783 T88 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 16783 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 16783 T88 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C5 name=collection1 org.apache.solr.core.SolrCore@42ed695f url=https://127.0.0.1:51590/collection1 node=127.0.0.1:51590_ C5_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:51590, core=collection1, node_name=127.0.0.1:51590_}
[junit4] 2> 16784 T88 C5 P51590 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:51590/collection1/
[junit4] 2> 16784 T88 C5 P51590 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 16784 T88 C5 P51590 oasc.SyncStrategy.syncToMe https://127.0.0.1:51590/collection1/ has no replicas
[junit4] 2> 16785 T88 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:51590/collection1/ shard3
[junit4] 2> 16785 T88 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 16786 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 16791 T43 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 16795 T43 oasc.DistributedQueue$LatchChildWatcher.pr
[...truncated too long message...]
ent.java:177)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:174)
[junit4] 2> at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:71)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:200)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:444)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:208)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 856304 T4875 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 856304 T4875 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 856318 T4793 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4] 2> 856370 T4793 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4] 2> 856371 T4793 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44075 44075
[junit4] 2> 857068 T4794 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44075 44075
[junit4] 2> 857168 T4880 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22b6f00f name:ZooKeeperConnection Watcher:127.0.0.1:44075/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
[junit4] 2> 857168 T4880 oascc.ConnectionManager.process Client->ZooKeeper status change trigger but we are already closed
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=UnloadDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=C7D26DC779D1943 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=bg -Dtests.timezone=PRT -Dtests.file.encoding=UTF-8
[junit4] FAILURE 48.2s J1 | UnloadDistributedZkTest.testDistribSearch <<<
[junit4] > Throwable #1: java.lang.AssertionError: Leader never changed
[junit4] > at __randomizedtesting.SeedInfo.seed([C7D26DC779D1943:8D9BA8C400C2797F]:0)
[junit4] > at org.apache.solr.cloud.UnloadDistributedZkTest.testCoreUnloadAndLeaders(UnloadDistributedZkTest.java:335)
[junit4] > at org.apache.solr.cloud.UnloadDistributedZkTest.doTest(UnloadDistributedZkTest.java:75)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 857179 T4793 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.UnloadDistributedZkTest-C7D26DC779D1943-001
[junit4] 2> 48168 T4792 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Lucene49: {other_tl1=PostingsFormat(name=Lucene41VarGapFixedInterval), _version_=PostingsFormat(name=Lucene41VarGapFixedInterval), multiDefault=FST41, a_t=PostingsFormat(name=Lucene41VarGapFixedInterval), intDefault=PostingsFormat(name=Lucene41VarGapFixedInterval), a_i1=PostingsFormat(name=Lucene41VarGapFixedInterval), id=Lucene41(blocksize=128), text=Pulsing41(freqCutoff=20 minBlockSize=88 maxBlockSize=243), range_facet_l=PostingsFormat(name=Lucene41VarGapFixedInterval), timestamp=PostingsFormat(name=Lucene41VarGapFixedInterval)}, docValues:{timestamp=DocValuesFormat(name=Disk)}, sim=DefaultSimilarity, locale=bg, timezone=PRT
[junit4] 2> NOTE: Linux 3.8.0-41-generic amd64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=8,threads=1,free=274108280,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [RankQueryTest, ChaosMonkeySafeLeaderTest, LegacyHTMLStripCharFilterTest, TestUpdate, OpenCloseCoreStressTest, DirectSolrSpellCheckerTest, HdfsCollectionsAPIDistributedZkTest, TestZkChroot, TestClassNameShortening, TestBinaryField, TestJmxIntegration, PathHierarchyTokenizerFactoryTest, TestComponentsName, TestFreeTextSuggestions, SpellCheckComponentTest, MultiTermTest, BadIndexSchemaTest, TestHighFrequencyDictionaryFactory, TestLRUCache, FullSolrCloudDistribCmdsTest, TestRangeQuery, TestAddFieldRealTimeGet, PingRequestHandlerTest, ZkNodePropsTest, NoFacetTest, TestIndexingPerformance, FileUtilsTest, CoreMergeIndexesAdminHandlerTest, UniqFieldsUpdateProcessorFactoryTest, TriLevelCompositeIdRoutingTest, TestImplicitCoreProperties, SortByFunctionTest, TestFastWriter, TestDistributedGrouping, DOMUtilTest, MinimalSchemaTest, SuggestComponentTest, TestSolrDeletionPolicy2, SolrCoreCheckLockOnStartupTest, DateFieldTest, OpenExchangeRatesOrgProviderTest, TestQueryTypes, TestFieldCollectionResource, StatsComponentTest, ResponseHeaderTest, TestMultiCoreConfBootstrap, TestStressLucene, TestFileDictionaryLookup, TestDefaultSearchFieldResource, RequiredFieldsTest, TestRequestStatusCollectionAPI, TestSchemaNameResource, TestCloudManagedSchema, TestExpandComponent, FieldFacetExtrasTest, DebugComponentTest, TestChildDocTransformer, CurrencyFieldXmlFileTest, AbstractAnalyticsStatsTest, DeleteReplicaTest, ParsingFieldUpdateProcessorsTest, TestManagedStopFilterFactory, TestDistribDocBasedVersion, TestNoOpRegenerator, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestCollapseQParserPlugin, DocValuesMultiTest, DocumentAnalysisRequestHandlerTest, CurrencyFieldOpenExchangeTest, BadCopyFieldTest, TestRecoveryHdfs, TestAnalyzedSuggestions, DistributedSuggestComponentTest, SpellCheckCollatorTest, CollectionsAPIAsyncDistributedZkTest, OutputWriterTest, FunctionTest, HdfsRecoveryZkTest, BlockDirectoryTest, TestMaxScoreQueryParser, CachingDirectoryFactoryTest, DistributedExpandComponentTest, TestSolrJ, SSLMigrationTest, SolrIndexConfigTest, TestSolrQueryParserDefaultOperatorResource, TestReplicationHandlerBackup, TestRecovery, ResourceLoaderTest, TestArbitraryIndexDir, MultiThreadedOCPTest, ZkSolrClientTest, StandardRequestHandlerTest, FieldAnalysisRequestHandlerTest, CoreContainerCoreInitFailuresTest, TestReRankQParserPlugin, TestSolrXMLSerializer, TestOmitPositions, BasicDistributedZk2Test, UnloadDistributedZkTest]
[junit4] Completed on J1 in 49.20s, 1 test, 1 failure <<< FAILURES!
[...truncated 641 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1296: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:920: There were test failures: 410 suites, 1693 tests, 1 error, 2 failures, 36 ignored (15 assumptions)
Total time: 59 minutes 42 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0_05 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any