You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/06/22 13:32:39 UTC
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_20-ea-b15) -
Build # 10627 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10627/
Java: 64bit/jdk1.8.0_20-ea-b15 -XX:+UseCompressedOops -XX:+UseG1GC
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard2 is not consistent. Got 73 from http://127.0.0.1:54227/xx_wpf/m/collection1lastClient and got 53 from http://127.0.0.1:45089/xx_wpf/m/collection1
Stack Trace:
java.lang.AssertionError: shard2 is not consistent. Got 73 from http://127.0.0.1:54227/xx_wpf/m/collection1lastClient and got 53 from http://127.0.0.1:45089/xx_wpf/m/collection1
at __randomizedtesting.SeedInfo.seed([3ED1087E3E2D6:81E56308F0BC82EA]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1131)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1110)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11228 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/init-core-data-001
[junit4] 2> 605970 T2114 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
[junit4] 2> 605971 T2114 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /xx_wpf/m
[junit4] 2> 605974 T2114 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 605974 T2114 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 605975 T2115 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 606075 T2114 oasc.ZkTestServer.run start zk server on port:58939
[junit4] 2> 606076 T2114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 606091 T2121 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77474d69 name:ZooKeeperConnection Watcher:127.0.0.1:58939 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 606091 T2114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 606091 T2114 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 606099 T2114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 606101 T2123 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69e0885 name:ZooKeeperConnection Watcher:127.0.0.1:58939/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 606101 T2114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 606101 T2114 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 606104 T2114 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 606105 T2114 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 606106 T2114 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 606108 T2114 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 606108 T2114 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 606110 T2114 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 606111 T2114 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 606113 T2114 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 606113 T2114 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 606115 T2114 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 606115 T2114 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 606117 T2114 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 606117 T2114 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 606118 T2114 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 606119 T2114 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 606123 T2114 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 606123 T2114 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 606126 T2114 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 606127 T2114 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 606130 T2114 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 606131 T2114 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 606134 T2114 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 606135 T2114 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 606138 T2114 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 606138 T2114 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 606383 T2114 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 606387 T2114 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37050
[junit4] 2> 606388 T2114 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 606389 T2114 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 606389 T2114 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-002
[junit4] 2> 606390 T2114 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-002/'
[junit4] 2> 606404 T2114 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-002/solr.xml
[junit4] 2> 606416 T2114 oasc.CoreContainer.<init> New CoreContainer 1162354057
[junit4] 2> 606416 T2114 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-002/]
[junit4] 2> 606417 T2114 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 606418 T2114 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 606418 T2114 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 606419 T2114 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 606419 T2114 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 606419 T2114 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 606419 T2114 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 606419 T2114 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 606420 T2114 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 606420 T2114 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 606420 T2114 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 606420 T2114 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 606421 T2114 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58939/solr
[junit4] 2> 606421 T2114 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 606422 T2114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 606424 T2134 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ad3e90d name:ZooKeeperConnection Watcher:127.0.0.1:58939 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 606426 T2114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 606429 T2114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 606431 T2136 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e6fbe82 name:ZooKeeperConnection Watcher:127.0.0.1:58939/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 606432 T2114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 606433 T2114 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 606437 T2114 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 606439 T2114 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 606441 T2114 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 606442 T2114 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 606444 T2114 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 606445 T2114 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37050_xx_wpf%2Fm
[junit4] 2> 606445 T2114 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37050_xx_wpf%2Fm
[junit4] 2> 606447 T2114 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 606448 T2114 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 606452 T2114 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:37050_xx_wpf%2Fm
[junit4] 2> 606453 T2114 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 606454 T2114 oasc.Overseer.start Overseer (id=91975567515975683-127.0.0.1:37050_xx_wpf%2Fm-n_0000000000) starting
[junit4] 2> 606456 T2114 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 606461 T2138 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 606461 T2114 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 606464 T2114 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 606465 T2114 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 606466 T2137 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 606467 T2139 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 606467 T2139 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 606468 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 606468 T2139 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 606469 T2137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 606469 T2137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:37050/xx_wpf/m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:37050_xx_wpf%2Fm",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 606469 T2137 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 606470 T2137 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 606471 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 606529 T2136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 607468 T2139 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 607469 T2139 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 607469 T2139 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 607469 T2139 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 607470 T2139 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 607470 T2139 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-002/collection1/'
[junit4] 2> 607471 T2139 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 607471 T2139 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 607471 T2139 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 607490 T2139 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 607498 T2139 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 607499 T2139 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 607502 T2139 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 607591 T2139 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 607592 T2139 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 607593 T2139 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 607595 T2139 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 607606 T2139 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 607607 T2139 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 607607 T2139 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/control/data/
[junit4] 2> 607607 T2139 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2378b3
[junit4] 2> 607608 T2139 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/control/data
[junit4] 2> 607608 T2139 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/control/data/index/
[junit4] 2> 607609 T2139 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 607610 T2139 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/control/data/index
[junit4] 2> 607610 T2139 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=15, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 607632 T2139 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 607633 T2139 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 607635 T2139 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 607635 T2139 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 607635 T2139 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 607636 T2139 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 607636 T2139 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 607636 T2139 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 607636 T2139 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 607636 T2139 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 607637 T2139 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 607637 T2139 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 607637 T2139 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 607637 T2139 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 607637 T2139 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 607638 T2139 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 607638 T2139 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 607638 T2139 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 607638 T2139 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 607640 T2139 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 607641 T2139 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 607641 T2139 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 607642 T2139 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=19, maxMergedSegmentMB=58.0849609375, floorSegmentMB=0.98828125, forceMergeDeletesPctAllowed=26.88757826516691, segmentsPerTier=48.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2507715702136052
[junit4] 2> 607642 T2139 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 607643 T2139 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 607643 T2139 oass.SolrIndexSearcher.<init> Opening Searcher@3b0be449[collection1] main
[junit4] 2> 607643 T2139 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 607644 T2139 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 607644 T2139 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 607644 T2139 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 607644 T2139 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 607644 T2139 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 607645 T2139 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 607645 T2139 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 607645 T2139 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 607645 T2139 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 607647 T2140 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3b0be449[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 607648 T2139 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 607648 T2143 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37050/xx_wpf/m collection:control_collection shard:shard1
[junit4] 2> 607649 T2114 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 607649 T2114 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 607649 T2143 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 607650 T2114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 607652 T2145 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49d53183 name:ZooKeeperConnection Watcher:127.0.0.1:58939/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 607652 T2114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 607653 T2114 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 607654 T2143 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 607655 T2114 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 607655 T2143 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 607656 T2143 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C696 name=collection1 org.apache.solr.core.SolrCore@76a04e15 url=http://127.0.0.1:37050/xx_wpf/m/collection1 node=127.0.0.1:37050_xx_wpf%2Fm C696_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:37050/xx_wpf/m, core=collection1, node_name=127.0.0.1:37050_xx_wpf%2Fm}
[junit4] 2> 607656 T2143 C696 P37050 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:37050/xx_wpf/m/collection1/
[junit4] 2> 607656 T2143 C696 P37050 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 607656 T2143 C696 P37050 oasc.SyncStrategy.syncToMe http://127.0.0.1:37050/xx_wpf/m/collection1/ has no replicas
[junit4] 2> 607656 T2143 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:37050/xx_wpf/m/collection1/ shard1
[junit4] 2> 607657 T2143 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 607657 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 607658 T2137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 607662 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 607667 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 607769 T2136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 607769 T2145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 607813 T2143 oasc.ZkController.register We are http://127.0.0.1:37050/xx_wpf/m/collection1/ and leader is http://127.0.0.1:37050/xx_wpf/m/collection1/
[junit4] 2> 607813 T2143 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37050/xx_wpf/m
[junit4] 2> 607813 T2143 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 607814 T2143 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 607814 T2143 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 607815 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 607815 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 607815 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 607815 T2143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 607816 T2137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 607817 T2137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:37050/xx_wpf/m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:37050_xx_wpf%2Fm",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 607821 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 607883 T2114 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 607884 T2114 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 607885 T2114 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49161
[junit4] 2> 607886 T2114 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 607886 T2114 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 607887 T2114 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-003
[junit4] 2> 607887 T2114 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-003/'
[junit4] 2> 607898 T2114 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-003/solr.xml
[junit4] 2> 607905 T2114 oasc.CoreContainer.<init> New CoreContainer 1218804622
[junit4] 2> 607905 T2114 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-003/]
[junit4] 2> 607906 T2114 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 607906 T2114 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 607906 T2114 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 607906 T2114 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 607907 T2114 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 607907 T2114 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 607907 T2114 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 607907 T2114 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 607907 T2114 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 607908 T2114 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 607908 T2114 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 607908 T2114 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 607909 T2114 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58939/solr
[junit4] 2> 607909 T2114 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 607909 T2114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 607911 T2156 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@493758c7 name:ZooKeeperConnection Watcher:127.0.0.1:58939 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 607912 T2114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 607914 T2114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 607916 T2158 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22ed99ea name:ZooKeeperConnection Watcher:127.0.0.1:58939/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 607917 T2114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 607922 T2114 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 607922 T2136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 607922 T2145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 608924 T2114 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49161_xx_wpf%2Fm
[junit4] 2> 608926 T2114 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49161_xx_wpf%2Fm
[junit4] 2> 608932 T2159 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 608932 T2159 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 608933 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 608933 T2159 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 608933 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 608934 T2137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 608934 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 608934 T2137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:49161/xx_wpf/m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:49161_xx_wpf%2Fm",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 608934 T2137 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 608935 T2137 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 608936 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 609038 T2145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 609038 T2158 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 609038 T2136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 609934 T2159 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 609934 T2159 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 609935 T2159 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 609935 T2159 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 609936 T2159 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 609937 T2159 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-003/collection1/'
[junit4] 2> 609938 T2159 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 609938 T2159 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 609938 T2159 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 609973 T2159 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 609984 T2159 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 609984 T2159 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 609988 T2159 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 610075 T2159 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 610076 T2159 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 610076 T2159 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 610079 T2159 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 610090 T2159 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 610091 T2159 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 610091 T2159 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty1/
[junit4] 2> 610091 T2159 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2378b3
[junit4] 2> 610092 T2159 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty1
[junit4] 2> 610092 T2159 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty1/index/
[junit4] 2> 610092 T2159 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 610093 T2159 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty1/index
[junit4] 2> 610094 T2159 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=15, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 610122 T2159 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 610123 T2159 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 610126 T2159 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 610127 T2159 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 610127 T2159 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 610127 T2159 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 610127 T2159 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 610128 T2159 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 610128 T2159 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 610128 T2159 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 610128 T2159 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 610129 T2159 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 610129 T2159 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 610129 T2159 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 610130 T2159 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 610130 T2159 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 610131 T2159 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 610131 T2159 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 610132 T2159 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 610133 T2159 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 610134 T2159 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 610135 T2159 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 610135 T2159 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=19, maxMergedSegmentMB=58.0849609375, floorSegmentMB=0.98828125, forceMergeDeletesPctAllowed=26.88757826516691, segmentsPerTier=48.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2507715702136052
[junit4] 2> 610136 T2159 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 610136 T2159 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 610137 T2159 oass.SolrIndexSearcher.<init> Opening Searcher@1b945659[collection1] main
[junit4] 2> 610137 T2159 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 610138 T2159 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 610138 T2159 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 610138 T2159 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 610138 T2159 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 610139 T2159 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 610139 T2159 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 610139 T2159 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 610139 T2159 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 610140 T2159 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 610141 T2160 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1b945659[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 610143 T2159 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 610143 T2163 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49161/xx_wpf/m collection:collection1 shard:shard2
[junit4] 2> 610144 T2114 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 610144 T2114 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 610145 T2163 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 610153 T2163 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 610154 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 610154 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 610154 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 610154 T2163 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 610155 T2163 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 610155 T2137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C697 name=collection1 org.apache.solr.core.SolrCore@71346f31 url=http://127.0.0.1:49161/xx_wpf/m/collection1 node=127.0.0.1:49161_xx_wpf%2Fm C697_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:49161/xx_wpf/m, core=collection1, node_name=127.0.0.1:49161_xx_wpf%2Fm}
[junit4] 2> 610155 T2163 C697 P49161 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49161/xx_wpf/m/collection1/
[junit4] 2> 610156 T2163 C697 P49161 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 610156 T2163 C697 P49161 oasc.SyncStrategy.syncToMe http://127.0.0.1:49161/xx_wpf/m/collection1/ has no replicas
[junit4] 2> 610156 T2163 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49161/xx_wpf/m/collection1/ shard2
[junit4] 2> 610156 T2163 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 610157 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 610159 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 610161 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 610264 T2136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 610264 T2158 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 610264 T2145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 610310 T2163 oasc.ZkController.register We are http://127.0.0.1:49161/xx_wpf/m/collection1/ and leader is http://127.0.0.1:49161/xx_wpf/m/collection1/
[junit4] 2> 610311 T2163 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49161/xx_wpf/m
[junit4] 2> 610311 T2163 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 610311 T2163 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 610311 T2163 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 610312 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 610313 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 610313 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 610314 T2137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 610314 T2163 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 610315 T2137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:49161/xx_wpf/m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:49161_xx_wpf%2Fm",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 610319 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 610396 T2114 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 610397 T2114 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 610398 T2114 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35088
[junit4] 2> 610400 T2114 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 610400 T2114 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 610401 T2114 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-004
[junit4] 2> 610401 T2114 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-004/'
[junit4] 2> 610414 T2114 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-004/solr.xml
[junit4] 2> 610421 T2114 oasc.CoreContainer.<init> New CoreContainer 25601231
[junit4] 2> 610421 T2114 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-004/]
[junit4] 2> 610422 T2114 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 610422 T2114 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 610424 T2114 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 610424 T2158 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 610424 T2114 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 610424 T2145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 610424 T2136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 610425 T2114 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 610425 T2114 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 610426 T2114 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 610426 T2114 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 610426 T2114 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 610427 T2114 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 610427 T2114 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 610427 T2114 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 610427 T2114 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58939/solr
[junit4] 2> 610428 T2114 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 610428 T2114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 610431 T2174 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@285fbd7e name:ZooKeeperConnection Watcher:127.0.0.1:58939 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 610433 T2114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 610436 T2114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 610438 T2176 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d5bb459 name:ZooKeeperConnection Watcher:127.0.0.1:58939/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 610439 T2114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 610444 T2114 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 611446 T2114 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35088_xx_wpf%2Fm
[junit4] 2> 611447 T2114 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35088_xx_wpf%2Fm
[junit4] 2> 611454 T2177 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 611454 T2177 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 611455 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 611455 T2177 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 611455 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 611455 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 611456 T2137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 611457 T2137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:35088/xx_wpf/m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:35088_xx_wpf%2Fm",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 611457 T2137 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 611457 T2137 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 611460 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 611569 T2176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 611570 T2158 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 611569 T2136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 611569 T2145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 612455 T2177 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 612456 T2177 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 612456 T2177 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 612456 T2177 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 612457 T2177 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 612457 T2177 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-004/collection1/'
[junit4] 2> 612458 T2177 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 612458 T2177 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 612459 T2177 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 612478 T2177 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 612488 T2177 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 612489 T2177 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 612495 T2177 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 612572 T2177 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 612573 T2177 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 612574 T2177 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 612576 T2177 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 612588 T2177 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 612588 T2177 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 612588 T2177 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty2/
[junit4] 2> 612589 T2177 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3f2378b3
[junit4] 2> 612589 T2177 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty2
[junit4] 2> 612590 T2177 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty2/index/
[junit4] 2> 612590 T2177 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 612591 T2177 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty2/index
[junit4] 2> 612592 T2177 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=15, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 612605 T2177 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 612605 T2177 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 612608 T2177 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 612608 T2177 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 612608 T2177 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 612609 T2177 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 612609 T2177 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 612609 T2177 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 612609 T2177 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 612609 T2177 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 612610 T2177 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 612610 T2177 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 612610 T2177 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 612610 T2177 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 612611 T2177 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 612611 T2177 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 612611 T2177 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 612612 T2177 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 612612 T2177 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 612613 T2177 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 612614 T2177 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 612615 T2177 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 612615 T2177 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=19, maxMergedSegmentMB=58.0849609375, floorSegmentMB=0.98828125, forceMergeDeletesPctAllowed=26.88757826516691, segmentsPerTier=48.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2507715702136052
[junit4] 2> 612616 T2177 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 612616 T2177 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 612616 T2177 oass.SolrIndexSearcher.<init> Opening Searcher@e9249d3[collection1] main
[junit4] 2> 612617 T2177 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 612618 T2177 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 612618 T2177 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 612618 T2177 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 612618 T2177 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 612619 T2177 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 612619 T2177 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 612619 T2177 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 612619 T2177 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 612619 T2177 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 612622 T2178 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e9249d3[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 612623 T2177 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 612624 T2181 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35088/xx_wpf/m collection:collection1 shard:shard3
[junit4] 2> 612625 T2114 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 612625 T2181 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 612625 T2114 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 612629 T2181 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 612630 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 612630 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 612630 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 612630 T2181 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 612631 T2181 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C698 name=collection1 org.apache.solr.core.SolrCore@6588b5ea url=http://127.0.0.1:35088/xx_wpf/m/collection1 node=127.0.0.1:35088_xx_wpf%2Fm C698_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:35088/xx_wpf/m, core=collection1, node_name=127.0.0.1:35088_xx_wpf%2Fm}
[junit4] 2> 612631 T2181 C698 P35088 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35088/xx_wpf/m/collection1/
[junit4] 2> 612631 T2181 C698 P35088 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 612631 T2181 C698 P35088 oasc.SyncStrategy.syncToMe http://127.0.0.1:35088/xx_wpf/m/collection1/ has no replicas
[junit4] 2> 612631 T2137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 612631 T2181 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35088/xx_wpf/m/collection1/ shard3
[junit4] 2> 612631 T2181 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 612633 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 612634 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 612635 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 612738 T2136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 612738 T2158 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 612738 T2145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 612738 T2176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 612785 T2181 oasc.ZkController.register We are http://127.0.0.1:35088/xx_wpf/m/collection1/ and leader is http://127.0.0.1:35088/xx_wpf/m/collection1/
[junit4] 2> 612785 T2181 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35088/xx_wpf/m
[junit4] 2> 612785 T2181 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 612785 T2181 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 612785 T2181 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 612788 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 612789 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 612789 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 612789 T2181 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 612789 T2137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 612790 T2137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:35088/xx_wpf/m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:35088_xx_wpf%2Fm",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 612791 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 612866 T2114 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 612868 T2114 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 612869 T2114 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52640
[junit4] 2> 612871 T2114 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 612872 T2114 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 612872 T2114 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-005
[junit4] 2> 612872 T2114 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-005/'
[junit4] 2> 612886 T2114 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-005/solr.xml
[junit4] 2> 612893 T2176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 612893 T2136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 612893 T2145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 612893 T2158 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 612894 T2114 oasc.CoreContainer.<init> New CoreContainer 1308138594
[junit4] 2> 612894 T2114 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-005/]
[junit4] 2> 612895 T2114 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 612895 T2114 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 612896 T2114 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 612896 T2114 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 612896 T2114 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 612896 T2114 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 612897 T2114 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 612897 T2114 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 612897 T2114 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 612898 T2114 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 612898 T2114 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 612898 T2114 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 612898 T2114 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58939/solr
[junit4] 2> 612899 T2114 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 612899 T2114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 612901 T2192 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ea51370 name:ZooKeeperConnection Watcher:127.0.0.1:58939 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 612902 T2114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 612906 T2114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 612907 T2194 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30dc59d9 name:ZooKeeperConnection Watcher:127.0.0.1:58939/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 612908 T2114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 612912 T2114 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 613914 T2114 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52640_xx_wpf%2Fm
[junit4] 2> 613915 T2114 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52640_xx_wpf%2Fm
[junit4] 2> 613922 T2195 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 613922 T2195 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 613924 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 613924 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 613924 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 613924 T2195 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 613925 T2137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 613925 T2137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:52640/xx_wpf/m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:52640_xx_wpf%2Fm",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 613925 T2137 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 613925 T2137 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 613927 T2136 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 614099 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 614100 T2158 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 614100 T2136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 614100 T2145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has o
[...truncated too long message...]
2> 659710 T2536 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 659711 T2536 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty10/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty10/index;done=false>>]
[junit4] 2> 659711 T2536 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty10/index
[junit4] 2> 659711 T2536 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty10 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty10;done=false>>]
[junit4] 2> 659712 T2536 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty10
[junit4] 2> 659712 T2536 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> ASYNC NEW_CORE C897 name=collection1 org.apache.solr.core.SolrCore@5d3184ca url=http://127.0.0.1:38339/xx_wpf/m/collection1 node=127.0.0.1:38339_xx_wpf%2Fm C897_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:38339/xx_wpf/m, core=collection1, node_name=127.0.0.1:38339_xx_wpf%2Fm}
[junit4] 2> 659773 T2483 C897 P38339 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
[junit4] 2> 659773 T2483 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
[junit4] 2> 659773 T2483 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@5d3184ca
[junit4] 2> 659775 T2483 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=3,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=58,cumulative_deletesById=29,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=7029,transaction_logs_total_number=4}
[junit4] 2> 659776 T2483 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 659776 T2483 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 659776 T2483 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 659779 T2483 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 659780 T2483 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 659780 T2483 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty11 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty11;done=false>>]
[junit4] 2> 659780 T2483 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty11
[junit4] 2> 659780 T2483 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty11/index.20140622171704619 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty11/index.20140622171704619;done=false>>]
[junit4] 2> 659781 T2483 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001/tempDir-001/jetty11/index.20140622171704619
[junit4] 2> 659781 T2483 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 659781 T2483 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:357)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-3ED1087E3E2D6-001
[junit4] 2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=383), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=383)), sim=DefaultSimilarity, locale=en_SG, timezone=Asia/Bishkek
[junit4] 2> NOTE: Linux 3.8.0-41-generic amd64/Oracle Corporation 1.8.0_20-ea (64-bit)/cpus=8,threads=1,free=196671984,total=525336576
[junit4] 2> NOTE: All tests run in this JVM: [FieldAnalysisRequestHandlerTest, OpenCloseCoreStressTest, TestFieldTypeResource, FullSolrCloudDistribCmdsTest, NoCacheHeaderTest, TestNonNRTOpen, TestRTGBase, PingRequestHandlerTest, TestAnalyzedSuggestions, TestBinaryField, DocumentAnalysisRequestHandlerTest, TestPartialUpdateDeduplication, TestFieldResource, BasicZkTest, ResourceLoaderTest, TestQueryTypes, TestFastWriter, TestSweetSpotSimilarityFactory, ChangedSchemaMergeTest, TestStressReorder, TestQuerySenderNoQuery, HighlighterTest, MergeStrategyTest, NoFacetTest, HighlighterConfigTest, AliasIntegrationTest, TestRandomFaceting, TestFileDictionaryLookup, TestLMDirichletSimilarityFactory, PluginInfoTest, TestHashPartitioner, SuggesterWFSTTest, TestReload, LeaderElectionIntegrationTest, TestReRankQParserPlugin, TestSerializedLuceneMatchVersion, BJQParserTest, TestQuerySenderListener, AlternateDirectoryTest, TestPhraseSuggestions, TestAddFieldRealTimeGet, DirectSolrConnectionTest, UniqFieldsUpdateProcessorFactoryTest, TestFastOutputStream, AtomicUpdatesTest, DisMaxRequestHandlerTest, TestModifyConfFiles, ShardRoutingCustomTest, SolrCmdDistributorTest, TestFreeTextSuggestions, DirectUpdateHandlerTest, TestArbitraryIndexDir, TestDistributedSearch, SimplePostToolTest, TestRandomMergePolicy, TestUniqueKeyFieldResource, MultiTermTest, DeleteInactiveReplicaTest, BlockCacheTest, TestStressLucene, SolrIndexConfigTest, DateMathParserTest, MultiThreadedOCPTest, DateRangeFieldTest, TestBinaryResponseWriter, PrimUtilsTest, ChaosMonkeySafeLeaderTest]
[junit4] Completed on J1 in 54.46s, 1 test, 1 failure <<< FAILURES!
[...truncated 755 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1296: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:920: There were test failures: 415 suites, 1703 tests, 1 failure, 36 ignored (15 assumptions)
Total time: 63 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0_20-ea-b15 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_05) - Build #
10628 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10628/
Java: 64bit/jdk1.8.0_05 -XX:+UseCompressedOops -XX:+UseParallelGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([1F9C9114B99738CE]:0)
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([1F9C9114B99738CE]:0)
Build Log:
[...truncated 12009 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/init-core-data-001
[junit4] 2> 377866 T1442 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 377866 T1442 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /p_/p
[junit4] 2> 377889 T1442 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 377890 T1442 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 377910 T1443 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 378009 T1442 oasc.ZkTestServer.run start zk server on port:39046
[junit4] 2> 378025 T1442 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 378053 T1449 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@635d41f3 name:ZooKeeperConnection Watcher:127.0.0.1:39046 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 378054 T1442 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 378054 T1442 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 378089 T1442 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 378089 T1451 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61a6312b name:ZooKeeperConnection Watcher:127.0.0.1:39046/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 378091 T1442 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 378091 T1442 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 378093 T1442 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 378094 T1442 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 378096 T1442 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 378097 T1442 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 378098 T1442 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 378100 T1442 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 378101 T1442 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 378103 T1442 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 378104 T1442 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 378106 T1442 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 378106 T1442 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 378108 T1442 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 378109 T1442 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 378111 T1442 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 378112 T1442 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 378114 T1442 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 378114 T1442 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 378116 T1442 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 378116 T1442 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 378118 T1442 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 378119 T1442 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 378121 T1442 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 378121 T1442 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 378123 T1442 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 378123 T1442 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 378157 T1442 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 378169 T1453 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@221e49b7 name:ZooKeeperConnection Watcher:127.0.0.1:39046/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 378170 T1442 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 378709 T1442 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 378716 T1442 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 378725 T1442 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:33781
[junit4] 2> 378727 T1442 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 378728 T1442 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 378728 T1442 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-002
[junit4] 2> 378729 T1442 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-002/'
[junit4] 2> 378747 T1442 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-002/solr.xml
[junit4] 2> 378757 T1442 oasc.CoreContainer.<init> New CoreContainer 1422404476
[junit4] 2> 378758 T1442 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-002/]
[junit4] 2> 378759 T1442 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 378759 T1442 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 378759 T1442 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 378760 T1442 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 378760 T1442 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 378760 T1442 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 378761 T1442 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 378761 T1442 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 378761 T1442 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 378764 T1442 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 378765 T1442 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 378765 T1442 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 378765 T1442 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39046/solr
[junit4] 2> 378766 T1442 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 378767 T1442 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 378768 T1464 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@192fa74b name:ZooKeeperConnection Watcher:127.0.0.1:39046 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 378768 T1442 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 378789 T1442 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 378791 T1466 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f4f0864 name:ZooKeeperConnection Watcher:127.0.0.1:39046/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 378791 T1442 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 378792 T1442 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 378794 T1442 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 378797 T1442 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 378800 T1442 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 378803 T1442 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 378805 T1442 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 378807 T1442 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33781_p_%2Fp
[junit4] 2> 378807 T1442 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33781_p_%2Fp
[junit4] 2> 378809 T1442 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 378811 T1442 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 378813 T1442 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:33781_p_%2Fp
[junit4] 2> 378814 T1442 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 378815 T1442 oasc.Overseer.start Overseer (id=91976108997672964-127.0.0.1:33781_p_%2Fp-n_0000000000) starting
[junit4] 2> 378818 T1442 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 378825 T1467 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 378836 T1442 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 378836 T1468 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 378838 T1442 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 378840 T1442 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 378845 T1469 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 378845 T1469 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 378846 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 378847 T1467 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 378848 T1467 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:33781/p_/p",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:33781_p_%2Fp",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 378848 T1467 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 378849 T1467 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 378847 T1469 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 378850 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 378856 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 379851 T1469 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 379851 T1469 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 379852 T1469 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 379852 T1469 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 379853 T1469 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 379853 T1469 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-002/collection1/'
[junit4] 2> 379854 T1469 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 379854 T1469 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 379855 T1469 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 379885 T1469 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 379896 T1469 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 379897 T1469 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 379903 T1469 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 380030 T1469 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 380032 T1469 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 380033 T1469 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 380036 T1469 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 380054 T1469 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 380054 T1469 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 380067 T1469 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/control/data/
[junit4] 2> 380067 T1469 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c4dfdcd
[junit4] 2> 380068 T1469 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/control/data
[junit4] 2> 380068 T1469 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/control/data/index/
[junit4] 2> 380069 T1469 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 380069 T1469 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/control/data/index
[junit4] 2> 380070 T1469 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=65.7138671875, floorSegmentMB=0.64453125, forceMergeDeletesPctAllowed=5.903296980545898, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 380074 T1469 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 380074 T1469 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 380078 T1469 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 380078 T1469 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 380079 T1469 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 380079 T1469 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 380079 T1469 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 380079 T1469 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 380080 T1469 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 380080 T1469 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 380080 T1469 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 380081 T1469 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 380081 T1469 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 380081 T1469 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 380081 T1469 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 380082 T1469 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 380082 T1469 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 380082 T1469 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 380083 T1469 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 380085 T1469 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 380087 T1469 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 380087 T1469 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 380088 T1469 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=60.4501953125, floorSegmentMB=1.634765625, forceMergeDeletesPctAllowed=28.162689063639753, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6813429138298107
[junit4] 2> 380089 T1469 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 380089 T1469 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 380089 T1469 oass.SolrIndexSearcher.<init> Opening Searcher@4bf26ab3[collection1] main
[junit4] 2> 380090 T1469 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 380090 T1469 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 380091 T1469 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 380091 T1469 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 380091 T1469 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 380092 T1469 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 380092 T1469 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 380092 T1469 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 380093 T1469 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 380093 T1469 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 380096 T1469 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 380097 T1470 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4bf26ab3[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 380109 T1473 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:33781/p_/p collection:control_collection shard:shard1
[junit4] 2> 380110 T1442 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 380110 T1442 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 380114 T1473 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 380118 T1473 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 380122 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 380123 T1467 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 380134 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 380135 T1473 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 380135 T1473 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C366 name=collection1 org.apache.solr.core.SolrCore@72d57de5 url=https://127.0.0.1:33781/p_/p/collection1 node=127.0.0.1:33781_p_%2Fp C366_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:33781/p_/p, core=collection1, node_name=127.0.0.1:33781_p_%2Fp}
[junit4] 2> 380135 T1473 C366 P33781 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:33781/p_/p/collection1/
[junit4] 2> 380135 T1473 C366 P33781 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 380135 T1473 C366 P33781 oasc.SyncStrategy.syncToMe https://127.0.0.1:33781/p_/p/collection1/ has no replicas
[junit4] 2> 380136 T1473 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:33781/p_/p/collection1/ shard1
[junit4] 2> 380136 T1473 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 380139 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 380139 T1442 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 380140 T1475 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e241f8d name:ZooKeeperConnection Watcher:127.0.0.1:39046/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 380140 T1442 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 380141 T1442 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 380141 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 380142 T1442 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 380243 T1475 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 380243 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 380290 T1473 oasc.ZkController.register We are https://127.0.0.1:33781/p_/p/collection1/ and leader is https://127.0.0.1:33781/p_/p/collection1/
[junit4] 2> 380291 T1473 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:33781/p_/p
[junit4] 2> 380291 T1473 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 380291 T1473 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 380291 T1473 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 380292 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 380292 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 380292 T1473 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 380293 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 380293 T1467 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 380294 T1467 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:33781/p_/p",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:33781_p_%2Fp",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 380297 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 380399 T1475 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 380399 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 380586 T1442 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 380587 T1442 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 380591 T1442 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 380593 T1442 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:37541
[junit4] 2> 380604 T1442 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 380605 T1442 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 380605 T1442 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-003
[junit4] 2> 380605 T1442 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-003/'
[junit4] 2> 380626 T1442 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-003/solr.xml
[junit4] 2> 380636 T1442 oasc.CoreContainer.<init> New CoreContainer 165948720
[junit4] 2> 380637 T1442 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-003/]
[junit4] 2> 380637 T1442 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 380638 T1442 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 380638 T1442 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 380638 T1442 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 380639 T1442 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 380639 T1442 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 380639 T1442 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 380639 T1442 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 380640 T1442 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 380643 T1442 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 380643 T1442 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 380643 T1442 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 380644 T1442 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39046/solr
[junit4] 2> 380644 T1442 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 380646 T1442 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 380648 T1486 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3cc484b9 name:ZooKeeperConnection Watcher:127.0.0.1:39046 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 380648 T1442 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 380657 T1442 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 380668 T1488 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50b43c90 name:ZooKeeperConnection Watcher:127.0.0.1:39046/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 380668 T1442 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 380672 T1442 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 381674 T1442 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37541_p_%2Fp
[junit4] 2> 381675 T1442 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37541_p_%2Fp
[junit4] 2> 381694 T1489 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 381694 T1489 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 381695 T1489 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 381695 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 381695 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 381696 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 381696 T1467 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 381697 T1467 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:37541/p_/p",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:37541_p_%2Fp",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 381697 T1467 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 381697 T1467 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 381699 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 381800 T1475 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 381800 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 381800 T1488 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 382695 T1489 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 382696 T1489 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 382696 T1489 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 382696 T1489 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 382697 T1489 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 382697 T1489 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-003/collection1/'
[junit4] 2> 382698 T1489 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 382698 T1489 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 382699 T1489 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 382725 T1489 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 382768 T1489 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 382769 T1489 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 382775 T1489 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 382914 T1489 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 382916 T1489 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 382917 T1489 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 382920 T1489 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 382933 T1489 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 382934 T1489 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 382934 T1489 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/jetty1/
[junit4] 2> 382934 T1489 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c4dfdcd
[junit4] 2> 382935 T1489 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/jetty1
[junit4] 2> 382935 T1489 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/jetty1/index/
[junit4] 2> 382936 T1489 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 382936 T1489 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/jetty1/index
[junit4] 2> 382937 T1489 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=65.7138671875, floorSegmentMB=0.64453125, forceMergeDeletesPctAllowed=5.903296980545898, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 382944 T1489 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 382944 T1489 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 382948 T1489 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 382948 T1489 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 382948 T1489 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 382949 T1489 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 382949 T1489 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 382949 T1489 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 382949 T1489 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 382950 T1489 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 382950 T1489 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 382950 T1489 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 382951 T1489 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 382951 T1489 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 382951 T1489 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 382952 T1489 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 382952 T1489 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 382952 T1489 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 382953 T1489 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 382955 T1489 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 382970 T1489 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 382970 T1489 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 382971 T1489 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=60.4501953125, floorSegmentMB=1.634765625, forceMergeDeletesPctAllowed=28.162689063639753, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6813429138298107
[junit4] 2> 382973 T1489 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 382973 T1489 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 382973 T1489 oass.SolrIndexSearcher.<init> Opening Searcher@2b353a96[collection1] main
[junit4] 2> 382973 T1489 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 382974 T1489 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 382975 T1489 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 382975 T1489 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 382975 T1489 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 382975 T1489 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 382976 T1489 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 382976 T1489 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 382976 T1489 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 382976 T1489 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 382979 T1490 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2b353a96[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 382981 T1489 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 382981 T1493 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:37541/p_/p collection:collection1 shard:shard2
[junit4] 2> 382982 T1442 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 382982 T1442 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 382985 T1493 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 383017 T1493 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 383018 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 383018 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 383019 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 383019 T1493 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 383019 T1493 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C367 name=collection1 org.apache.solr.core.SolrCore@409d060c url=https://127.0.0.1:37541/p_/p/collection1 node=127.0.0.1:37541_p_%2Fp C367_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:37541/p_/p, core=collection1, node_name=127.0.0.1:37541_p_%2Fp}
[junit4] 2> 383019 T1493 C367 P37541 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:37541/p_/p/collection1/
[junit4] 2> 383020 T1493 C367 P37541 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 383020 T1493 C367 P37541 oasc.SyncStrategy.syncToMe https://127.0.0.1:37541/p_/p/collection1/ has no replicas
[junit4] 2> 383020 T1493 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:37541/p_/p/collection1/ shard2
[junit4] 2> 383020 T1493 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 383021 T1467 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 383023 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 383024 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 383026 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 383128 T1475 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 383128 T1488 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 383128 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 383178 T1493 oasc.ZkController.register We are https://127.0.0.1:37541/p_/p/collection1/ and leader is https://127.0.0.1:37541/p_/p/collection1/
[junit4] 2> 383178 T1493 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:37541/p_/p
[junit4] 2> 383178 T1493 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 383178 T1493 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 383178 T1493 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 383180 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 383181 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 383181 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 383181 T1493 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 383182 T1467 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 383182 T1467 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:37541/p_/p",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:37541_p_%2Fp",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 383185 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 383289 T1475 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 383289 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 383289 T1488 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 383346 T1442 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 383347 T1442 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 383364 T1442 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 383365 T1442 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:49735
[junit4] 2> 383367 T1442 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 383368 T1442 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 383372 T1442 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-004
[junit4] 2> 383372 T1442 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-004/'
[junit4] 2> 383394 T1442 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-004/solr.xml
[junit4] 2> 383418 T1442 oasc.CoreContainer.<init> New CoreContainer 93196862
[junit4] 2> 383419 T1442 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-004/]
[junit4] 2> 383420 T1442 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 383420 T1442 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 383420 T1442 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 383421 T1442 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 383421 T1442 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 383421 T1442 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 383422 T1442 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 383422 T1442 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 383422 T1442 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 383436 T1442 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 383437 T1442 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 383437 T1442 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 383438 T1442 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39046/solr
[junit4] 2> 383438 T1442 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 383462 T1442 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 383462 T1504 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e5003cb name:ZooKeeperConnection Watcher:127.0.0.1:39046 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 383462 T1442 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 383482 T1442 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 383484 T1506 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c00456c name:ZooKeeperConnection Watcher:127.0.0.1:39046/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 383484 T1442 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 383489 T1442 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 384491 T1442 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49735_p_%2Fp
[junit4] 2> 384492 T1442 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49735_p_%2Fp
[junit4] 2> 384510 T1507 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 384510 T1507 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 384511 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 384511 T1507 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 384511 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 384512 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 384514 T1467 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 384514 T1467 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:49735/p_/p",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:49735_p_%2Fp",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 384515 T1467 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 384515 T1467 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 384516 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 384622 T1475 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 384622 T1488 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 384622 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 384622 T1506 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 385512 T1507 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 385512 T1507 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 385513 T1507 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 385513 T1507 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 385513 T1507 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 385513 T1507 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-004/collection1/'
[junit4] 2> 385514 T1507 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 385515 T1507 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 385515 T1507 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 385556 T1507 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 385566 T1507 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 385574 T1507 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 385578 T1507 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 385663 T1507 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 385665 T1507 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 385666 T1507 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 385668 T1507 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 385687 T1507 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 385688 T1507 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 385688 T1507 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/jetty2/
[junit4] 2> 385689 T1507 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c4dfdcd
[junit4] 2> 385689 T1507 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/jetty2
[junit4] 2> 385690 T1507 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/jetty2/index/
[junit4] 2> 385690 T1507 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 385690 T1507 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/jetty2/index
[junit4] 2> 385692 T1507 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=65.7138671875, floorSegmentMB=0.64453125, forceMergeDeletesPctAllowed=5.903296980545898, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 385695 T1507 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 385696 T1507 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 385699 T1507 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 385700 T1507 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 385700 T1507 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 385700 T1507 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 385700 T1507 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 385701 T1507 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 385701 T1507 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 385701 T1507 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 385702 T1507 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 385702 T1507 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 385703 T1507 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 385703 T1507 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 385703 T1507 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 385703 T1507 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 385704 T1507 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 385718 T1507 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 385718 T1507 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 385721 T1507 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 385722 T1507 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 385722 T1507 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 385723 T1507 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=60.4501953125, floorSegmentMB=1.634765625, forceMergeDeletesPctAllowed=28.162689063639753, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6813429138298107
[junit4] 2> 385724 T1507 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 385724 T1507 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 385725 T1507 oass.SolrIndexSearcher.<init> Opening Searcher@7a85b89e[collection1] main
[junit4] 2> 385725 T1507 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 385725 T1507 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 385726 T1507 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 385726 T1507 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 385726 T1507 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 385726 T1507 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 385727 T1507 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 385727 T1507 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 385727 T1507 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 385727 T1507 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 385729 T1508 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7a85b89e[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 385731 T1507 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 385731 T1511 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:49735/p_/p collection:collection1 shard:shard3
[junit4] 2> 385731 T1442 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 385732 T1442 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 385738 T1511 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 385745 T1511 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 385746 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 385746 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 385746 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 385747 T1467 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 385748 T1511 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 385748 T1511 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C368 name=collection1 org.apache.solr.core.SolrCore@411d600e url=https://127.0.0.1:49735/p_/p/collection1 node=127.0.0.1:49735_p_%2Fp C368_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:49735/p_/p, core=collection1, node_name=127.0.0.1:49735_p_%2Fp}
[junit4] 2> 385748 T1511 C368 P49735 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:49735/p_/p/collection1/
[junit4] 2> 385749 T1511 C368 P49735 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 385749 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 385750 T1511 C368 P49735 oasc.SyncStrategy.syncToMe https://127.0.0.1:49735/p_/p/collection1/ has no replicas
[junit4] 2> 385750 T1511 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:49735/p_/p/collection1/ shard3
[junit4] 2> 385750 T1511 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 385754 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 385756 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 385858 T1475 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 385858 T1488 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 385858 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 385858 T1506 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 385906 T1511 oasc.ZkController.register We are https://127.0.0.1:49735/p_/p/collection1/ and leader is https://127.0.0.1:49735/p_/p/collection1/
[junit4] 2> 385906 T1511 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:49735/p_/p
[junit4] 2> 385906 T1511 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 385907 T1511 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 385907 T1511 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 385910 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 385910 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 385910 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 385910 T1511 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 385911 T1467 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 385911 T1467 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:49735/p_/p",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:49735_p_%2Fp",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 385913 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 386014 T1475 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 386015 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 386014 T1488 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 386014 T1506 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 386089 T1442 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 386090 T1442 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 386219 T1442 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 386220 T1442 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:59550
[junit4] 2> 386223 T1442 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 386223 T1442 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 386223 T1442 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-005
[junit4] 2> 386225 T1442 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-005/'
[junit4] 2> 386272 T1442 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-005/solr.xml
[junit4] 2> 386282 T1442 oasc.CoreContainer.<init> New CoreContainer 374109106
[junit4] 2> 386282 T1442 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-005/]
[junit4] 2> 386283 T1442 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 386283 T1442 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 386284 T1442 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 386284 T1442 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 386284 T1442 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 386285 T1442 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 386285 T1442 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 386286 T1442 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 386286 T1442 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 386288 T1442 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 386289 T1442 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 386309 T1442 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 386310 T1442 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39046/solr
[junit4] 2> 386310 T1442 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 386312 T1442 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 386313 T1522 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54cfa36c name:ZooKeeperConnection Watcher:127.0.0.1:39046 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 386313 T1442 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 386317 T1442 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 386317 T1524 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41922ccd name:ZooKeeperConnection Watcher:127.0.0.1:39046/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 386318 T1442 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 386332 T1442 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 387334 T1442 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59550_p_%2Fp
[junit4] 2> 387335 T1442 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59550_p_%2Fp
[junit4] 2> 387342 T1525 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 387342 T1525 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 387344 T1525 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 387344 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 387344 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 387344 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 387346 T1467 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 387346 T1467 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:59550/p_/p",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:59550_p_%2Fp",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 387346 T1467 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 387346 T1467 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 387348 T1466 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 387519 T1506 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 387519 T1475 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 387519 T1488 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 387519 T1524 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 387519 T1466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 388344 T1525 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 388344 T1525 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 388345 T1525 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 388345 T1525 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 388346 T1525 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 388346 T1525 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-005/collection1/'
[junit4] 2> 388347 T1525 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-005/collection1/lib/.svn/' to classloader
[junit4] 2> 388347 T1525 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-005/collection1/lib/classes/' to classloader
[junit4] 2> 388348 T1525 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-1F9C9114B99738CE-001/tempDir-005/collection1/lib/README' to classloader
[junit4] 2> 388375 T1525 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 388387 T1525 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 388388 T1525 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 388394 T1525 o
[...truncated too long message...]
t.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
[junit4] 2> at org.apache.solr.cloud.LeaderInitiatedRecoveryThread.sendRecoveryCommandWithRetry(LeaderInitiatedRecoveryThread.java:117)
[junit4] 2> at org.apache.solr.cloud.LeaderInitiatedRecoveryThread.run(LeaderInitiatedRecoveryThread.java:74)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 71) Thread[id=2532, name=qtp352287972-2532 Acceptor1 SslSelectChannelConnector@127.0.0.1:33453, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)
[junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 72) Thread[id=2601, name=qtp897388045-2601 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
[junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
[junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4] 2> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 73) Thread[id=2378, name=qtp2134593490-2378 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
[junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
[junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4] 2> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 74) Thread[id=2545, name=coreZkRegister-1425-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 75) Thread[id=2603, name=qtp897388045-2603 Acceptor1 SslSelectChannelConnector@127.0.0.1:39413, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:956)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 7603106 T2539 oaz.ClientCnxn$SendThread.run WARN Session 0x146c3c8a8a1005f for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[junit4] 2>
[junit4] 2> 7603114 T2610 oaz.ClientCnxn$SendThread.run WARN Session 0x146c3c8a8a10065 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[junit4] 2>
[junit4] 2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=475), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=475)), sim=DefaultSimilarity, locale=hr_HR, timezone=Etc/GMT+6
[junit4] 2> NOTE: Linux 3.8.0-41-generic amd64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=8,threads=89,free=94777944,total=337117184
[junit4] 2> NOTE: All tests run in this JVM: [TestUniqueKeyFieldResource, TestComponentsName, DistributedExpandComponentTest, TestSolrXmlPersistor, TestExpandComponent, TestFiltering, OutputWriterTest, TestOverriddenPrefixQueryForCustomFieldType, DistribDocExpirationUpdateProcessorTest, CursorMarkTest, HdfsBasicDistributedZkTest, TestSolr4Spatial, MultiThreadedOCPTest, PingRequestHandlerTest, SpellCheckCollatorTest, TestIBSimilarityFactory, BlockDirectoryTest, TestRandomFaceting, DirectSolrSpellCheckerTest, DateMathParserTest, HdfsLockFactoryTest, DirectUpdateHandlerTest, FileUtilsTest, StatsComponentTest, TestCollectionAPI, TestBinaryResponseWriter, TestRandomMergePolicy, DistanceFunctionTest, TestReRankQParserPlugin, TestManagedStopFilterFactory, TestHashPartitioner, AddSchemaFieldsUpdateProcessorFactoryTest, SolrCmdDistributorTest, TestQuerySenderListener, CurrencyFieldOpenExchangeTest, HighlighterConfigTest, TestCollationField, TestFileDictionaryLookup, SSLMigrationTest, TestPartialUpdateDeduplication, SuggesterTSTTest, EnumFieldTest, HighlighterTest, LeaderElectionIntegrationTest, StandardRequestHandlerTest, TestQueryTypes, BlockCacheTest, TestSerializedLuceneMatchVersion, HdfsChaosMonkeySafeLeaderTest, TestUpdate, TestDFRSimilarityFactory, PrimUtilsTest, TestXIncludeConfig, TestFoldingMultitermQuery, JSONWriterTest, DocValuesMultiTest, ChaosMonkeySafeLeaderTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=1F9C9114B99738CE -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=hr_HR -Dtests.timezone=Etc/GMT+6 -Dtests.file.encoding=US-ASCII
[junit4] ERROR 0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
[junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4] > at __randomizedtesting.SeedInfo.seed([1F9C9114B99738CE]:0)
[junit4] Completed on J1 in 7225.31s, 1 test, 2 errors <<< FAILURES!
[...truncated 140 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1296: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:920: There were test failures: 415 suites, 1703 tests, 1 suite-level error, 1 error, 169 ignored (15 assumptions)
Total time: 163 minutes 0 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0_05 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any