You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/10/18 02:29:40 UTC
[JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4925 - Failure
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4925/
1 tests failed.
REGRESSION: org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch
Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:11201/gllb/wr, https://127.0.0.1:11187/gllb/wr, https://127.0.0.1:11195/gllb/wr, https://127.0.0.1:11171/gllb/wr, https://127.0.0.1:11578/gllb/wr]
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:11201/gllb/wr, https://127.0.0.1:11187/gllb/wr, https://127.0.0.1:11195/gllb/wr, https://127.0.0.1:11171/gllb/wr, https://127.0.0.1:11578/gllb/wr]
at __randomizedtesting.SeedInfo.seed([217FCCCCCF39A5F8:A09942D4B866C5C4]:0)
at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:333)
at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:1015)
at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:793)
at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:736)
at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:172)
at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:145)
at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:89)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not remove replica : delLiveColl/shard1/core_node1
at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:569)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:354)
at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:313)
... 47 more
Build Log:
[...truncated 12194 lines...]
[junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
[junit4] 2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/init-core-data-001
[junit4] 2> 1481160 T5086 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
[junit4] 2> 1481160 T5086 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /gllb/wr
[junit4] 2> 1481165 T5086 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 1481165 T5086 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1481166 T5087 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1481266 T5086 oasc.ZkTestServer.run start zk server on port:21570
[junit4] 2> 1481267 T5086 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1481268 T5086 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1481271 T5093 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e1b1c75 name:ZooKeeperConnection Watcher:127.0.0.1:21570 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1481271 T5086 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1481271 T5086 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1481272 T5086 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1481274 T5086 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1481275 T5086 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1481276 T5095 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38cede16 name:ZooKeeperConnection Watcher:127.0.0.1:21570/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1481276 T5086 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1481277 T5086 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1481277 T5086 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1481279 T5086 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1481280 T5086 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1481281 T5086 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1481283 T5086 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1481284 T5086 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1481286 T5086 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 1481287 T5086 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1481389 T5086 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1481389 T5086 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1481391 T5086 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1481392 T5086 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1481394 T5086 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1481394 T5086 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1481396 T5086 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1481397 T5086 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1481398 T5086 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1481399 T5086 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 1481401 T5086 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1481401 T5086 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1481403 T5086 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1481404 T5086 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1481405 T5086 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1481406 T5086 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1481408 T5086 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1481408 T5086 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1481410 T5086 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1481411 T5086 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1481412 T5097 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@574dc393 name:ZooKeeperConnection Watcher:127.0.0.1:21570/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1481412 T5086 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1481412 T5086 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1483568 T5086 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1483576 T5086 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1483578 T5086 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:11578
[junit4] 2> 1483581 T5086 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1483582 T5086 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1483582 T5086 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-002
[junit4] 2> 1483582 T5086 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-002/'
[junit4] 2> 1483645 T5086 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-002/solr.xml
[junit4] 2> 1483731 T5086 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-002
[junit4] 2> 1483732 T5086 oasc.CoreContainer.<init> New CoreContainer 1364411116
[junit4] 2> 1483732 T5086 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-002/]
[junit4] 2> 1483733 T5086 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1483734 T5086 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1483734 T5086 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1483734 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1483735 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 1483735 T5086 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1483735 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1483736 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1483736 T5086 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1483737 T5086 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1483740 T5086 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 1483742 T5086 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1483742 T5086 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1483743 T5086 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1483743 T5086 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:21570/solr
[junit4] 2> 1483743 T5086 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1483744 T5086 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1483745 T5086 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1483746 T5108 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b9e81dc name:ZooKeeperConnection Watcher:127.0.0.1:21570 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1483747 T5086 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1483748 T5086 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1483749 T5086 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1483750 T5110 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7420bc63 name:ZooKeeperConnection Watcher:127.0.0.1:21570/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1483751 T5086 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1483752 T5086 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1483753 T5086 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1483755 T5086 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1483757 T5086 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1483759 T5086 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1483761 T5086 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1483762 T5086 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:11578_gllb%2Fwr
[junit4] 2> 1483763 T5086 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:11578_gllb%2Fwr
[junit4] 2> 1483765 T5086 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1483766 T5086 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1483768 T5086 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1483769 T5086 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:11578_gllb%2Fwr
[junit4] 2> 1483770 T5086 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1483771 T5086 oasc.Overseer.start Overseer (id=92640996769595396-127.0.0.1:11578_gllb%2Fwr-n_0000000000) starting
[junit4] 2> 1483773 T5086 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1483780 T5086 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1483781 T5112 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1483782 T5086 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1483783 T5086 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1483784 T5086 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1483787 T5111 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1483790 T5086 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-002
[junit4] 2> 1485275 T5086 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-002/conf/
[junit4] 2> 1485532 T5086 oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 1485533 T5086 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4] 2> 1485533 T5086 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1485537 T5086 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1485538 T5086 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1485539 T5116 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37d1a75a name:ZooKeeperConnection Watcher:127.0.0.1:21570/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1485539 T5086 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1485539 T5086 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1485540 T5086 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1485541 T5086 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1485542 T5086 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
[junit4] 2> 1485542 T5086 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1485542 T5086 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1485543 T5118 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50fcea14 name:ZooKeeperConnection Watcher:127.0.0.1:21570/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1485543 T5086 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1485543 T5086 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1485566 T5119 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1485568 T5111 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
[junit4] 2> 1485568 T5111 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 1485569 T5111 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
[junit4] 2> 1485570 T5111 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
[junit4] 2> 1485571 T5119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1485572 T5120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1487192 T5086 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 1487193 T5086 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1487198 T5086 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1487201 T5086 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:11171
[junit4] 2> 1487204 T5086 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1487205 T5086 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1487205 T5086 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-003
[junit4] 2> 1487206 T5086 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-003/'
[junit4] 2> 1487258 T5086 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-003/solr.xml
[junit4] 2> 1487335 T5086 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-003
[junit4] 2> 1487335 T5086 oasc.CoreContainer.<init> New CoreContainer 135732258
[junit4] 2> 1487336 T5086 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-003/]
[junit4] 2> 1487337 T5086 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1487337 T5086 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1487338 T5086 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1487338 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1487338 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 1487339 T5086 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1487339 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1487339 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1487340 T5086 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1487340 T5086 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1487343 T5086 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 1487345 T5086 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1487345 T5086 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1487346 T5086 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1487346 T5086 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:21570/solr
[junit4] 2> 1487347 T5086 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1487347 T5086 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1487348 T5086 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1487349 T5131 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@faa02a8 name:ZooKeeperConnection Watcher:127.0.0.1:21570 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1487350 T5086 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1487351 T5086 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1487352 T5086 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1487353 T5133 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5017864a name:ZooKeeperConnection Watcher:127.0.0.1:21570/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1487353 T5086 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1487357 T5086 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1488361 T5086 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:11171_gllb%2Fwr
[junit4] 2> 1488362 T5086 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:11171_gllb%2Fwr
[junit4] 2> 1488365 T5086 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1488369 T5086 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-003
[junit4] 2> 1488373 T5086 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-003/conf/
[junit4] 2> 1489301 T5086 oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 1489302 T5086 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4] 2> 1489302 T5086 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1490524 T5086 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 1490525 T5086 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1490530 T5086 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1490531 T5086 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:11187
[junit4] 2> 1490533 T5086 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1490533 T5086 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1490534 T5086 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-004
[junit4] 2> 1490534 T5086 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-004/'
[junit4] 2> 1490576 T5086 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-004/solr.xml
[junit4] 2> 1490637 T5086 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-004
[junit4] 2> 1490638 T5086 oasc.CoreContainer.<init> New CoreContainer 1972637437
[junit4] 2> 1490638 T5086 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-004/]
[junit4] 2> 1490639 T5086 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1490639 T5086 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1490639 T5086 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1490640 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1490640 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 1490640 T5086 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1490640 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1490640 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1490641 T5086 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1490641 T5086 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1490643 T5086 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 1490645 T5086 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1490645 T5086 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1490645 T5086 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1490645 T5086 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:21570/solr
[junit4] 2> 1490646 T5086 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1490646 T5086 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1490646 T5086 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1490648 T5146 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5cf7caaf name:ZooKeeperConnection Watcher:127.0.0.1:21570 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1490649 T5086 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1490649 T5086 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1490650 T5086 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1490651 T5148 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@405449b9 name:ZooKeeperConnection Watcher:127.0.0.1:21570/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1490651 T5086 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1490655 T5086 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1491658 T5086 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:11187_gllb%2Fwr
[junit4] 2> 1491659 T5086 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:11187_gllb%2Fwr
[junit4] 2> 1491663 T5086 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1491667 T5086 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-004
[junit4] 2> 1493333 T5086 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-004/conf/
[junit4] 2> 1493334 T5086 oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 1493335 T5086 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4] 2> 1493335 T5086 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1494928 T5086 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 1494929 T5086 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1494933 T5086 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1494934 T5086 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:11195
[junit4] 2> 1494937 T5086 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1494937 T5086 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1494937 T5086 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-005
[junit4] 2> 1494938 T5086 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-005/'
[junit4] 2> 1494980 T5086 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-005/solr.xml
[junit4] 2> 1495040 T5086 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-005
[junit4] 2> 1495041 T5086 oasc.CoreContainer.<init> New CoreContainer 1744263985
[junit4] 2> 1495041 T5086 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-005/]
[junit4] 2> 1495042 T5086 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1495042 T5086 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1495043 T5086 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1495043 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1495043 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 1495043 T5086 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1495044 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1495044 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1495044 T5086 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1495044 T5086 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1495046 T5086 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 1495048 T5086 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1495048 T5086 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1495048 T5086 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1495049 T5086 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:21570/solr
[junit4] 2> 1495049 T5086 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1495049 T5086 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1495050 T5086 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1495051 T5161 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d912012 name:ZooKeeperConnection Watcher:127.0.0.1:21570 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1495052 T5086 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1495052 T5086 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1495054 T5086 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1495055 T5163 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23a59269 name:ZooKeeperConnection Watcher:127.0.0.1:21570/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1495055 T5086 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1495058 T5086 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1496061 T5086 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:11195_gllb%2Fwr
[junit4] 2> 1496062 T5086 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:11195_gllb%2Fwr
[junit4] 2> 1496065 T5086 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1496068 T5086 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-005
[junit4] 2> 1496081 T5086 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-005/conf/
[junit4] 2> 1496844 T5086 oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 1496845 T5086 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4] 2> 1496845 T5086 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1497836 T5086 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
[junit4] 2> 1497836 T5086 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1497841 T5086 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1497842 T5086 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:11201
[junit4] 2> 1497845 T5086 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1497845 T5086 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1497845 T5086 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-006
[junit4] 2> 1497846 T5086 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-006/'
[junit4] 2> 1497881 T5086 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-006/solr.xml
[junit4] 2> 1497931 T5086 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-006
[junit4] 2> 1497932 T5086 oasc.CoreContainer.<init> New CoreContainer 845854624
[junit4] 2> 1497932 T5086 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-006/]
[junit4] 2> 1497933 T5086 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1497933 T5086 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1497933 T5086 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1497933 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1497934 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 1497934 T5086 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1497934 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1497934 T5086 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1497934 T5086 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1497935 T5086 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1497937 T5086 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 1497938 T5086 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1497938 T5086 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1497939 T5086 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1497939 T5086 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:21570/solr
[junit4] 2> 1497939 T5086 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1497939 T5086 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1497940 T5086 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1497942 T5176 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23f0e5ef name:ZooKeeperConnection Watcher:127.0.0.1:21570 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1497943 T5086 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1497943 T5086 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1497945 T5086 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1497946 T5178 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@468a31a3 name:ZooKeeperConnection Watcher:127.0.0.1:21570/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1497946 T5086 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1497949 T5086 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1498953 T5086 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:11201_gllb%2Fwr
[junit4] 2> 1498954 T5086 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:11201_gllb%2Fwr
[junit4] 2> 1498957 T5086 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1498962 T5086 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-006
[junit4] 2> 1500412 T5086 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-006/conf/
[junit4] 2> 1500581 T5086 oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 1500582 T5086 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4] 2> 1500582 T5086 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1500588 T5086 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1500589 T5086 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1500590 T5182 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49f25609 name:ZooKeeperConnection Watcher:127.0.0.1:21570/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1500591 T5086 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1500591 T5086 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1500592 T5086 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1500594 T5086 oasc.AbstractFullDistribZkTestBase.createCollection Creating collection with stateFormat=2: delLiveColl
[junit4] 2> 1500594 T5086 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1500595 T5086 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1500596 T5184 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51a4016 name:ZooKeeperConnection Watcher:127.0.0.1:21570/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1500596 T5086 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1500597 T5086 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1500598 T5086 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1500734 T5106 oasha.CollectionsHandler.handleCreateAction Creating Collection : maxShardsPerNode=2&name=delLiveColl&action=CREATE&wt=javabin&numShards=2&replicationFactor=2&stateFormat=2&version=2
[junit4] 2> 1500735 T5119 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1500736 T5112 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
[junit4] 2> "operation":"create",
[junit4] 2> "fromApi":"true",
[junit4] 2> "name":"delLiveColl",
[junit4] 2> "replicationFactor":"2",
[junit4] 2> "numShards":"2",
[junit4] 2> "maxShardsPerNode":"2",
[junit4] 2> "stateFormat":"2"}
[junit4] 2> 1500737 T5186 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
[junit4] 2> "operation":"create",
[junit4] 2> "fromApi":"true",
[junit4] 2> "name":"delLiveColl",
[junit4] 2> "replicationFactor":"2",
[junit4] 2> "numShards":"2",
[junit4] 2> "maxShardsPerNode":"2",
[junit4] 2> "stateFormat":"2"}
[junit4] 2> 1500738 T5186 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
[junit4] 2> 1500738 T5186 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/delLiveColl
[junit4] 2> 1500738 T5186 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
[junit4] 2> 1500740 T5119 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1500742 T5111 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: delLiveColl
[junit4] 2> 1500742 T5111 oasc.Overseer$ClusterStateUpdater.createCollection Create collection delLiveColl with shards [shard1, shard2]
[junit4] 2> 1500742 T5111 oasc.Overseer$ClusterStateUpdater.createCollection state version delLiveColl 2
[junit4] 2> 1500744 T5111 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/delLiveColl/state.json
[junit4] 2> 1500745 T5134 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 1500745 T5164 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 1500745 T5188 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 1500745 T5179 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 1500745 T5187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 1500745 T5120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 1500745 T5119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 1500745 T5149 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 1500840 T5186 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection delLiveColl, shardNames [shard1, shard2] , replicationFactor : 2
[junit4] 2> 1500841 T5186 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica1 as part of slice shard1 of collection delLiveColl on 127.0.0.1:11578_gllb%2Fwr
[junit4] 2> 1500841 T5186 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica2 as part of slice shard1 of collection delLiveColl on 127.0.0.1:11171_gllb%2Fwr
[junit4] 2> 1500843 T5186 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica1 as part of slice shard2 of collection delLiveColl on 127.0.0.1:11187_gllb%2Fwr
[junit4] 2> 1500844 T5186 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica2 as part of slice shard2 of collection delLiveColl on 127.0.0.1:11195_gllb%2Fwr
[junit4] 2> 1500966 T5105 oasha.CoreAdminHandler.handleCreateAction core create command collection.configName=conf1&collection=delLiveColl&name=delLiveColl_shard1_replica1&action=CREATE&wt=javabin&numShards=2&qt=/admin/cores&version=2&shard=shard1
[junit4] 2> 1500968 T5156 oasha.CoreAdminHandler.handleCreateAction core create command collection.configName=conf1&collection=delLiveColl&name=delLiveColl_shard2_replica2&action=CREATE&wt=javabin&numShards=2&qt=/admin/cores&version=2&shard=shard2
[junit4] 2> 1500969 T5141 oasha.CoreAdminHandler.handleCreateAction core create command collection.configName=conf1&collection=delLiveColl&name=delLiveColl_shard2_replica1&action=CREATE&wt=javabin&numShards=2&qt=/admin/cores&version=2&shard=shard2
[junit4] 2> 1500971 T5105 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica1 state=down collection=delLiveColl
[junit4] 2> 1500971 T5156 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica2 state=down collection=delLiveColl
[junit4] 2> 1500972 T5119 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1500972 T5141 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica1 state=down collection=delLiveColl
[junit4] 2> 1500972 T5126 oasha.CoreAdminHandler.handleCreateAction core create command collection.configName=conf1&collection=delLiveColl&name=delLiveColl_shard1_replica2&action=CREATE&wt=javabin&numShards=2&qt=/admin/cores&version=2&shard=shard1
[junit4] 2> 1500972 T5105 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
[junit4] 2> 1500974 T5156 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
[junit4] 2> 1500974 T5141 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
[junit4] 2> 1500974 T5111 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "node_name":"127.0.0.1:11578_gllb%2Fwr",
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"delLiveColl",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:11578/gllb/wr",
[junit4] 2> "state":"down",
[junit4] 2> "core":"delLiveColl_shard1_replica1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1"}
[junit4] 2> 1500974 T5156 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
[junit4] 2> 1500975 T5141 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
[junit4] 2> 1500976 T5156 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper...
[junit4] 2> 1500976 T5126 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica2 state=down collection=delLiveColl
[junit4] 2> 1500976 T5141 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper...
[junit4] 2> 1500977 T5156 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0
[junit4] 2> 1500977 T5156 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1500977 T5141 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0
[junit4] 2> 1500978 T5141 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1500978 T5126 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
[junit4] 2> 1500978 T5126 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
[junit4] 2> 1500979 T5126 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper...
[junit4] 2> 1500979 T5111 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/delLiveColl/state.json
[junit4] 2> 1500980 T5149 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1500980 T5126 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1
[junit4] 2> 1500980 T5164 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1500980 T5134 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1500981 T5149 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1
[junit4] 2> 1500981 T5126 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1500981 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1
[junit4] 2> 1500981 T5164 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1
[junit4] 2> 1500981 T5105 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
[junit4] 2> 1500982 T5105 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper...
[junit4] 2> 1500983 T5105 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1
[junit4] 2> 1500984 T5105 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1500984 T5105 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica1
[junit4] 2> 1500984 T5105 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
[junit4] 2> 1500985 T5105 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1500985 T5105 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
[junit4] 2> 1500985 T5111 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1
[junit4] 2> 1500986 T5111 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "node_name":"127.0.0.1:11195_gllb%2Fwr",
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"delLiveColl",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:11195/gllb/wr",
[junit4] 2> "state":"down",
[junit4] 2> "core":"delLiveColl_shard2_replica2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard2"}
[junit4] 2> 1500986 T5105 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1500986 T5105 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-002/delLiveColl_shard1_replica1/'
[junit4] 2> 1500987 T5119 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1500987 T5111 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/delLiveColl/state.json
[junit4] 2> 1500988 T5119 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1500988 T5164 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1500988 T5149 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1500988 T5134 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1501027 T5164 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2
[junit4] 2> 1501028 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2
[junit4] 2> 1501028 T5119 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2
[junit4] 2> 1501027 T5149 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2
[junit4] 2> 1501030 T5111 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2
[junit4] 2> 1501030 T5111 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "node_name":"127.0.0.1:11187_gllb%2Fwr",
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"delLiveColl",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:11187/gllb/wr",
[junit4] 2> "state":"down",
[junit4] 2> "core":"delLiveColl_shard2_replica1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard2"}
[junit4] 2> 1501031 T5119 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1501032 T5111 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/delLiveColl/state.json
[junit4] 2> 1501032 T5119 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1501032 T5134 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1501032 T5149 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1501032 T5164 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1501033 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3
[junit4] 2> 1501033 T5164 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3
[junit4] 2> 1501033 T5149 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3
[junit4] 2> 1501034 T5119 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3
[junit4] 2> 1501036 T5111 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3
[junit4] 2> 1501037 T5111 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "node_name":"127.0.0.1:11171_gllb%2Fwr",
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"delLiveColl",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:11171/gllb/wr",
[junit4] 2> "state":"down",
[junit4] 2> "core":"delLiveColl_shard1_replica2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1"}
[junit4] 2> 1501038 T5119 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1501038 T5111 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/delLiveColl/state.json
[junit4] 2> 1501039 T5119 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1501039 T5149 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1501039 T5164 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1501039 T5134 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1501040 T5149 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 4
[junit4] 2> 1501040 T5164 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 4
[junit4] 2> 1501040 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 4
[junit4] 2> 1501040 T5119 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 4
[junit4] 2> 1501136 T5105 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 1501182 T5105 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1501183 T5105 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1501194 T5105 oass.IndexSchema.readSchema [delLiveColl_shard1_replica1] Schema name=test
[junit4] 2> 1501555 T5105 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1501565 T5105 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1501568 T5105 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1501577 T5105 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1501582 T5105 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1501586 T5105 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1501587 T5105 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1501589 T5105 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1501589 T5105 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1501590 T5105 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1501591 T5105 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1501591 T5105 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica1' using configuration from collection delLiveColl
[junit4] 2> 1501591 T5105 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1501592 T5105 oasc.SolrCore.<init> [delLiveColl_shard1_replica1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-002/delLiveColl_shard1_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-002/delLiveColl_shard1_replica1/data/
[junit4] 2> 1501592 T5105 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@62d962c
[junit4] 2> 1501593 T5105 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-002/delLiveColl_shard1_replica1/data
[junit4] 2> 1501594 T5105 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-002/delLiveColl_shard1_replica1/data/index/
[junit4] 2> 1501594 T5105 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-002/delLiveColl_shard1_replica1/data/index' doesn't exist. Creating new index...
[junit4] 2> 1501595 T5105 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-002/delLiveColl_shard1_replica1/data/index
[junit4] 2> 1501595 T5105 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=10, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 1501596 T5105 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-1e61dd79)),segFN=segments_1,generation=1}
[junit4] 2> 1501597 T5105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1501601 T5105 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1501601 T5105 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1501602 T5105 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1501602 T5105 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1501602 T5105 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1501603 T5105 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1501603 T5105 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1501604 T5105 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1501604 T5105 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1501605 T5105 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 1501605 T5105 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 1501606 T5105 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 1501606 T5105 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1501607 T5105 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1501607 T5105 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1501607 T5105 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1501608 T5105 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1501608 T5105 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1501609 T5105 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1501610 T5105 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 1501629 T5105 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1501632 T5105 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1501634 T5105 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1501636 T5105 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1501643 T5105 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1501643 T5105 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1501644 T5105 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=51.8427734375, floorSegmentMB=1.451171875, forceMergeDeletesPctAllowed=19.5665253061794, segmentsPerTier=6.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1501645 T5105 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-1e61dd79)),segFN=segments_1,generation=1}
[junit4] 2> 1501645 T5105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1501645 T5105 oass.SolrIndexSearcher.<init> Opening Searcher@6285fb97[delLiveColl_shard1_replica1] main
[junit4] 2> 1501646 T5105 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
[junit4] 2> 1501647 T5105 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1501647 T5105 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1501648 T5105 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1501648 T5105 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1501648 T5105 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1501649 T5105 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 1501649 T5105 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1501650 T5105 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
[junit4] 2> 1501651 T5105 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
[junit4] 2> 1501652 T5105 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1501652 T5105 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1501655 T5197 oasc.SolrCore.registerSearcher [delLiveColl_shard1_replica1] Registered new searcher Searcher@6285fb97[delLiveColl_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1501658 T5105 oasc.CoreContainer.registerCore registering core: delLiveColl_shard1_replica1
[junit4] 2> 1501659 T5105 oasc.ZkController.register Register replica - core:delLiveColl_shard1_replica1 address:https://127.0.0.1:11578/gllb/wr collection:delLiveColl shard:shard1
[junit4] 2> 1501660 T5105 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard1/election
[junit4] 2> 1501664 T5105 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1501665 T5119 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1501665 T5105 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=7.4973518E15ms
[junit4] 2> 1501667 T5111 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 4
[junit4] 2> 1501669 T5111 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/delLiveColl/state.json
[junit4] 2> 1501670 T5119 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1501670 T5134 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1501670 T5164 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1501670 T5149 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating...
[junit4] 2> 1501671 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 5
[junit4] 2> 1501671 T5149 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 5
[junit4] 2> 1501671 T5164 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 5
[junit4] 2> 1501671 T5119 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 5
[junit4] 2> 1501978 T5156 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica2
[junit4] 2> 1501978 T5141 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica1
[junit4] 2> 1501979 T5156 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
[junit4] 2> 1501979 T5141 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
[junit4] 2> 1501980 T5156 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1501980 T5141 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1501980 T5156 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
[junit4] 2> 1501981 T5141 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
[junit4] 2> 1501982 T5156 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1501982 T5156 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-005/delLiveColl_shard2_replica2/'
[junit4] 2> 1501982 T5141 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1501982 T5126 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica2
[junit4] 2> 1501983 T5141 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-004/delLiveColl_shard2_replica1/'
[junit4] 2> 1501983 T5126 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
[junit4] 2> 1501984 T5126 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1501984 T5126 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
[junit4] 2> 1501985 T5126 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1501985 T5126 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001/tempDir-003/delLiveColl_shard1_replica2/'
[junit4] 2> 1502049 T5156 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 1502054 T5141 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 1502058 T5126 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 1502101 T5156 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1502105 T5141 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1502109 T5126 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1502202 T5156 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1502206 T5141 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1502210 T5126 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1502213 T5156 oass.IndexSchema.readSchema [delLiveColl_shard2_replica2] Schema name=test
[junit4] 2> 1502218 T5141 oass.IndexSchema.readSchema [delLiveColl_shard2_replica1] Schema name=test
[junit4] 2> 1502221 T5126 oass.IndexSchema.readSchema [delLiveColl_shard1_replica2] Schema name=test
[junit4] 2> 1502652 T5156 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1502656 T5141 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1502659 T5156 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1502662 T5156 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1502662 T5141 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1502666 T5141 oass.IndexSchema.readSchema uniq
[...truncated too long message...]
2> 1699734 T5271 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:11195_gllb%2Fwr
[junit4] 2> 1699738 T5258 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:11201_gllb%2Fwr
[junit4] 2> 1699738 T5258 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1699739 T5258 oasc.Overseer.start Overseer (id=92640996769595406-127.0.0.1:11201_gllb%2Fwr-n_0000000004) starting
[junit4] 2> 1699745 T5258 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1699745 T5277 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1699746 T5276 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1701236 T5164 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1701238 T5086 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/gllb/wr,null}
[junit4] 2> 1706096 T5086 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 11201
[junit4] 2> 1706097 T5086 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=845854624
[junit4] 2> 1706099 T5086 oasc.Overseer.close Overseer (id=92640996769595406-127.0.0.1:11201_gllb%2Fwr-n_0000000004) closing
[junit4] 2> 1706099 T5276 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:11201_gllb%2Fwr
[junit4] 2> 1706102 T5258 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1706103 T5086 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/gllb/wr,null}
[junit4] 2> 1708185 T5086 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4] 2> 1708187 T5086 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:21570 21570
[junit4] 2> 1708449 T5087 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:21570 21570
[junit4] 2> 1708451 T5087 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 8 /solr/aliases.json
[junit4] 2> 8 /solr/clusterstate.json
[junit4] 2> 4 /solr/collections/delLiveColl/state.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 8 /solr/live_nodes
[junit4] 2> 5 /solr/overseer/collection-queue-work
[junit4] 2> 5 /solr/overseer/queue
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=DeleteReplicaTest -Dtests.method=testDistribSearch -Dtests.seed=217FCCCCCF39A5F8 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=lv -Dtests.timezone=US/Hawaii -Dtests.file.encoding=UTF-8
[junit4] ERROR 227s J0 | DeleteReplicaTest.testDistribSearch <<<
[junit4] > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:11201/gllb/wr, https://127.0.0.1:11187/gllb/wr, https://127.0.0.1:11195/gllb/wr, https://127.0.0.1:11171/gllb/wr, https://127.0.0.1:11578/gllb/wr]
[junit4] > at __randomizedtesting.SeedInfo.seed([217FCCCCCF39A5F8:A09942D4B866C5C4]:0)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:333)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:1015)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:793)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:736)
[junit4] > at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:172)
[junit4] > at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:145)
[junit4] > at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:89)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not remove replica : delLiveColl/shard1/core_node1
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:569)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:354)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:313)
[junit4] > ... 47 more
[junit4] 2> 1708474 T5086 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-217FCCCCCF39A5F8-001
[junit4] 2> 227319 T5085 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene50): {}, docValues:{}, sim=DefaultSimilarity, locale=lv, timezone=US/Hawaii
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=127362992,total=299892736
[junit4] 2> NOTE: All tests run in this JVM: [TestCollectionAPI, TestCustomSort, BadComponentTest, TestRawResponseWriter, TestCloudManagedSchema, MinimalSchemaTest, TestArbitraryIndexDir, TestCoreContainer, ConnectionManagerTest, TestStressReorder, TestSolrJ, SuggesterWFSTTest, TestDefaultSimilarityFactory, ZkSolrClientTest, TestLazyCores, ShardRoutingCustomTest, BlockDirectoryTest, CacheHeaderTest, TestSolrXml, MultiThreadedOCPTest, TestUtils, TestBinaryResponseWriter, AlternateDirectoryTest, SchemaVersionSpecificBehaviorTest, TestIBSimilarityFactory, TestOrdValues, TestFaceting, DistributedFacetPivotSmallTest, TestTrie, HdfsBasicDistributedZkTest, BasicDistributedZkTest, DirectSolrConnectionTest, ResourceLoaderTest, ChangedSchemaMergeTest, ConvertedLegacyTest, TestRequestStatusCollectionAPI, DistributedQueryElevationComponentTest, TestAnalyzedSuggestions, TestSolrDeletionPolicy1, DistanceFunctionTest, BlockCacheTest, TestRangeQuery, TestFoldingMultitermQuery, TestWriterPerf, TestReloadAndDeleteDocs, TestPivotHelperCode, TestSolrIndexConfig, TestNRTOpen, SpellingQueryConverterTest, TestExpandComponent, SoftAutoCommitTest, ChaosMonkeyNothingIsSafeTest, TestFieldTypeCollectionResource, TestDefaultSearchFieldResource, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, LeaderElectionIntegrationTest, RecoveryZkTest, TestRandomFaceting, TestZkChroot, TestRandomDVFaceting, ZkCLITest, DistributedSpellCheckComponentTest, SpellCheckComponentTest, TestGroupingSearch, TestSort, TestFiltering, BasicFunctionalityTest, OverseerCollectionProcessorTest, AnalysisAfterCoreReloadTest, TestExtendedDismaxParser, SuggesterTest, TestUpdate, TestAtomicUpdateErrorCases, QueryEqualityTest, TestQueryUtils, DirectSolrSpellCheckerTest, PrimitiveFieldTypeTest, DocumentBuilderTest, TermVectorComponentTest, FastVectorHighlighterTest, TestJmxIntegration, MBeansHandlerTest, JsonLoaderTest, PingRequestHandlerTest, CoreContainerCoreInitFailuresTest, TestQuerySenderListener, ResponseLogComponentTest, TestSolrDeletionPolicy2, TestSearchPerf, TestElisionMultitermQuery, TestFuzzyAnalyzedSuggestions, TestPostingsSolrHighlighter, TestLuceneMatchVersion, TestCodecSupport, SynonymTokenizerTest, EchoParamsTest, TestPerFieldSimilarity, TestFastWriter, OpenExchangeRatesOrgProviderTest, TestFastLRUCache, PreAnalyzedFieldTest, DateFieldTest, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, CollectionsAPIAsyncDistributedZkTest, DeleteReplicaTest]
[junit4] Completed on J0 in 227.86s, 1 test, 1 error <<< FAILURES!
[...truncated 300 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:524: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:961: There were test failures: 437 suites, 1794 tests, 1 error, 58 ignored (22 assumptions)
Total time: 81 minutes 46 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-Tests-trunk-Java7 #4924
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 156714285 bytes
Compression is 0.0%
Took 29 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4928 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4928/
1 tests failed.
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
expected:<0> but was:<1>
Stack Trace:
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([40325B5E2A1DE510:C1D4D5465D42852C]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:153)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11214 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/init-core-data-001
[junit4] 2> 482759 T1118 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 482759 T1118 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 482766 T1118 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 482767 T1118 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 482768 T1119 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 482868 T1118 oasc.ZkTestServer.run start zk server on port:44305
[junit4] 2> 482869 T1118 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 482870 T1118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 482874 T1125 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6249481c name:ZooKeeperConnection Watcher:127.0.0.1:44305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 482874 T1118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 482875 T1118 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 482875 T1118 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 482878 T1118 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 482879 T1118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 482881 T1127 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9335f4c name:ZooKeeperConnection Watcher:127.0.0.1:44305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 482881 T1118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 482881 T1118 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 482882 T1118 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 482884 T1118 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 482886 T1118 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 482888 T1118 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 482890 T1118 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 482891 T1118 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 482895 T1118 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 482895 T1118 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 482998 T1118 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 482999 T1118 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 483001 T1118 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 483002 T1118 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 483004 T1118 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 483005 T1118 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 483007 T1118 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 483008 T1118 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 483010 T1118 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 483011 T1118 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 483013 T1118 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 483014 T1118 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 483016 T1118 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 483017 T1118 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 483019 T1118 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 483020 T1118 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 483022 T1118 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 483022 T1118 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 484393 T1118 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 484397 T1118 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:44312
[junit4] 2> 484397 T1118 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 484398 T1118 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 484398 T1118 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-002
[junit4] 2> 484398 T1118 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-002/'
[junit4] 2> 484429 T1118 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-002/solr.xml
[junit4] 2> 484484 T1118 oasc.CoreContainer.<init> New CoreContainer 1937877811
[junit4] 2> 484484 T1118 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-002/]
[junit4] 2> 484485 T1118 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 484485 T1118 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 484485 T1118 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 484486 T1118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 484486 T1118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 484486 T1118 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 484486 T1118 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 484486 T1118 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 484487 T1118 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 484487 T1118 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 484487 T1118 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 484488 T1118 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 484488 T1118 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 484488 T1118 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 484488 T1118 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44305/solr
[junit4] 2> 484489 T1118 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 484489 T1118 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 484490 T1118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 484491 T1138 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@460edc83 name:ZooKeeperConnection Watcher:127.0.0.1:44305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 484492 T1118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 484492 T1118 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 484495 T1118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 484496 T1140 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f700d33 name:ZooKeeperConnection Watcher:127.0.0.1:44305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 484496 T1118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 484497 T1118 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 484500 T1118 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 484502 T1118 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 484504 T1118 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 484506 T1118 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 484509 T1118 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 484510 T1118 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44312_
[junit4] 2> 484511 T1118 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44312_
[junit4] 2> 484513 T1118 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 484515 T1118 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 484516 T1118 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 484518 T1118 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44312_
[junit4] 2> 484518 T1118 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 484520 T1118 oasc.Overseer.start Overseer (id=92654515837468675-127.0.0.1:44312_-n_0000000000) starting
[junit4] 2> 484522 T1118 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 484529 T1118 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 484529 T1142 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 484530 T1118 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 484532 T1118 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 484533 T1118 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 484536 T1141 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 484541 T1144 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 484541 T1144 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 484542 T1144 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 484542 T1145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 484545 T1141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "node_name":"127.0.0.1:44312_",
[junit4] 2> "roles":null,
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://127.0.0.1:44312",
[junit4] 2> "state":"down"}
[junit4] 2> 484545 T1141 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 484545 T1141 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
[junit4] 2> 484545 T1141 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 484548 T1145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 485542 T1144 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 485543 T1144 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 485544 T1144 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 485544 T1144 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 485545 T1144 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 485545 T1144 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-002/collection1/'
[junit4] 2> 485547 T1144 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 485548 T1144 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 485548 T1144 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 485620 T1144 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 485671 T1144 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 485772 T1144 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 485782 T1144 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 486056 T1144 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 486058 T1144 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 486059 T1144 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 486063 T1144 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 486080 T1144 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 486080 T1144 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 486131 T1144 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/control/data/
[junit4] 2> 486131 T1144 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ac846a2
[junit4] 2> 486134 T1144 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/control/data
[junit4] 2> 486134 T1144 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/control/data/index/
[junit4] 2> 486135 T1144 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 486136 T1144 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/control/data/index
[junit4] 2> 486136 T1144 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=32, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 486141 T1144 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 486141 T1144 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 486144 T1144 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 486144 T1144 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 486145 T1144 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 486145 T1144 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 486145 T1144 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 486145 T1144 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 486146 T1144 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 486146 T1144 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 486146 T1144 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 486146 T1144 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 486147 T1144 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 486147 T1144 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 486147 T1144 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 486148 T1144 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 486148 T1144 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 486148 T1144 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 486148 T1144 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 486148 T1144 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 486149 T1144 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 486149 T1144 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 486163 T1144 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 486164 T1144 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 486166 T1144 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 486167 T1144 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 486170 T1144 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 486170 T1144 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 486171 T1144 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21149206554905187]
[junit4] 2> 486172 T1144 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 486172 T1144 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 486172 T1144 oass.SolrIndexSearcher.<init> Opening Searcher@3e4d5e76[collection1] main
[junit4] 2> 486172 T1144 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 486173 T1144 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 486174 T1144 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 486174 T1144 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 486174 T1144 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 486174 T1144 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 486175 T1144 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 486175 T1144 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 486176 T1144 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
[junit4] 2> 486177 T1144 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
[junit4] 2> 486178 T1144 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 486178 T1144 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 486180 T1146 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3e4d5e76[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 486182 T1144 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 486182 T1149 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44312 collection:control_collection shard:shard1
[junit4] 2> 486183 T1118 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4] 2> 486183 T1118 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 486183 T1149 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 486184 T1118 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 486184 T1118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 486186 T1151 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f586234 name:ZooKeeperConnection Watcher:127.0.0.1:44305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 486186 T1118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 486186 T1118 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 486188 T1118 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 486189 T1149 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 486191 T1145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 486191 T1149 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 486191 T1118 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 486191 T1149 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C306 name=collection1 org.apache.solr.core.SolrCore@1eaeb43 url=http://127.0.0.1:44312/collection1 node=127.0.0.1:44312_ C306_STATE=coll:control_collection core:collection1 props:{core=collection1, node_name=127.0.0.1:44312_, base_url=http://127.0.0.1:44312, state=down}
[junit4] 2> 486192 T1149 C306 P44312 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44312/collection1/
[junit4] 2> 486192 T1149 C306 P44312 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 486192 T1149 C306 P44312 oasc.SyncStrategy.syncToMe http://127.0.0.1:44312/collection1/ has no replicas
[junit4] 2> 486192 T1149 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44312/collection1/ shard1
[junit4] 2> 486192 T1149 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 486195 T1145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 486195 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 486201 T1145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 486202 T1145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 486202 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 486248 T1149 oasc.ZkController.register We are http://127.0.0.1:44312/collection1/ and leader is http://127.0.0.1:44312/collection1/
[junit4] 2> 486249 T1149 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44312
[junit4] 2> 486249 T1149 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 486249 T1149 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 486249 T1149 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 486250 T1145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 486253 T1141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "node_name":"127.0.0.1:44312_",
[junit4] 2> "roles":null,
[junit4] 2> "numShards":"2",
[junit4] 2> "base_url":"http://127.0.0.1:44312",
[junit4] 2> "state":"active"}
[junit4] 2> 486356 T1145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 486356 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 487826 T1118 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 487827 T1118 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 487830 T1118 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:44318
[junit4] 2> 487831 T1118 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 487831 T1118 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 487832 T1118 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-003
[junit4] 2> 487832 T1118 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-003/'
[junit4] 2> 487881 T1118 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-003/solr.xml
[junit4] 2> 487971 T1118 oasc.CoreContainer.<init> New CoreContainer 1489468631
[junit4] 2> 487971 T1118 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-003/]
[junit4] 2> 487973 T1118 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 487973 T1118 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 487973 T1118 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 487974 T1118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 487974 T1118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 487975 T1118 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 487975 T1118 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 487975 T1118 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 487976 T1118 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 487976 T1118 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 487977 T1118 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 487977 T1118 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 487977 T1118 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 487978 T1118 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 487978 T1118 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44305/solr
[junit4] 2> 487978 T1118 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 487979 T1118 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 487980 T1118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 487981 T1163 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10d534ca name:ZooKeeperConnection Watcher:127.0.0.1:44305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 487982 T1118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 487982 T1118 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 487984 T1118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 487985 T1165 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3dc024c4 name:ZooKeeperConnection Watcher:127.0.0.1:44305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 487986 T1118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 487991 T1118 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 488994 T1118 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44318_
[junit4] 2> 488995 T1118 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44318_
[junit4] 2> 488999 T1118 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 489006 T1167 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 489006 T1167 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 489007 T1145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 489007 T1167 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 489010 T1141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "node_name":"127.0.0.1:44318_",
[junit4] 2> "roles":null,
[junit4] 2> "numShards":"2",
[junit4] 2> "base_url":"http://127.0.0.1:44318",
[junit4] 2> "state":"down"}
[junit4] 2> 489010 T1141 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 489010 T1141 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 1
[junit4] 2> 489010 T1141 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 489012 T1145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 489013 T1166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 489013 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 490007 T1167 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 490008 T1167 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 490009 T1167 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 490009 T1167 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 490010 T1167 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 490010 T1167 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-003/collection1/'
[junit4] 2> 490012 T1167 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 490014 T1167 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 490015 T1167 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 490089 T1167 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 490141 T1167 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 490242 T1167 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 490250 T1167 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 490532 T1167 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 490534 T1167 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 490535 T1167 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 490539 T1167 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 490556 T1167 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 490556 T1167 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 490557 T1167 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-003/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty1/
[junit4] 2> 490557 T1167 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ac846a2
[junit4] 2> 490558 T1167 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty1
[junit4] 2> 490559 T1167 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty1/index/
[junit4] 2> 490560 T1167 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 490561 T1167 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty1/index
[junit4] 2> 490562 T1167 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=32, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 490566 T1167 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 490566 T1167 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 490570 T1167 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 490570 T1167 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 490571 T1167 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 490571 T1167 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 490571 T1167 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 490571 T1167 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 490572 T1167 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 490572 T1167 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 490572 T1167 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 490572 T1167 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 490573 T1167 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 490573 T1167 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 490574 T1167 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 490574 T1167 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 490574 T1167 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 490574 T1167 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 490575 T1167 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 490575 T1167 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 490575 T1167 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 490576 T1167 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 490597 T1167 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 490599 T1167 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 490600 T1167 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 490602 T1167 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 490605 T1167 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 490605 T1167 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 490606 T1167 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21149206554905187]
[junit4] 2> 490607 T1167 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 490608 T1167 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 490608 T1167 oass.SolrIndexSearcher.<init> Opening Searcher@5c3e794b[collection1] main
[junit4] 2> 490608 T1167 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 490609 T1167 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 490609 T1167 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 490610 T1167 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 490610 T1167 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 490610 T1167 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 490611 T1167 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 490611 T1167 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 490611 T1167 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 490611 T1167 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 490614 T1168 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5c3e794b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 490617 T1167 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 490618 T1171 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44318 collection:collection1 shard:shard2
[junit4] 2> 490618 T1118 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4] 2> 490618 T1118 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 490619 T1171 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 490623 T1171 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 490625 T1145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 490625 T1171 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 490625 T1171 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C307 name=collection1 org.apache.solr.core.SolrCore@61c8ef13 url=http://127.0.0.1:44318/collection1 node=127.0.0.1:44318_ C307_STATE=coll:collection1 core:collection1 props:{core=collection1, node_name=127.0.0.1:44318_, base_url=http://127.0.0.1:44318, state=down}
[junit4] 2> 490626 T1171 C307 P44318 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44318/collection1/
[junit4] 2> 490626 T1171 C307 P44318 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 490626 T1171 C307 P44318 oasc.SyncStrategy.syncToMe http://127.0.0.1:44318/collection1/ has no replicas
[junit4] 2> 490626 T1171 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44318/collection1/ shard2
[junit4] 2> 490626 T1171 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 490630 T1145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 490630 T1166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 490630 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 490635 T1145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 490636 T1145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 490636 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 490636 T1166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 490681 T1171 oasc.ZkController.register We are http://127.0.0.1:44318/collection1/ and leader is http://127.0.0.1:44318/collection1/
[junit4] 2> 490682 T1171 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44318
[junit4] 2> 490682 T1171 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 490682 T1171 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 490682 T1171 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 490683 T1145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 490685 T1141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "node_name":"127.0.0.1:44318_",
[junit4] 2> "roles":null,
[junit4] 2> "numShards":"2",
[junit4] 2> "base_url":"http://127.0.0.1:44318",
[junit4] 2> "state":"active"}
[junit4] 2> 490789 T1166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 490789 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 490789 T1145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 491884 T1118 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 491884 T1118 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 491887 T1118 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:44330
[junit4] 2> 491887 T1118 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 491887 T1118 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 491888 T1118 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-004
[junit4] 2> 491888 T1118 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-004/'
[junit4] 2> 491921 T1118 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-004/solr.xml
[junit4] 2> 491980 T1118 oasc.CoreContainer.<init> New CoreContainer 1954062971
[junit4] 2> 491980 T1118 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-004/]
[junit4] 2> 491981 T1118 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 491981 T1118 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 491981 T1118 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 491982 T1118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 491982 T1118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 491982 T1118 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 491982 T1118 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 491982 T1118 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 491983 T1118 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 491983 T1118 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 491983 T1118 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 491984 T1118 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 491984 T1118 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 491984 T1118 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 491984 T1118 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44305/solr
[junit4] 2> 491985 T1118 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 491985 T1118 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 491986 T1118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 491987 T1182 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bf374fa name:ZooKeeperConnection Watcher:127.0.0.1:44305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 491987 T1118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 491988 T1118 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 491989 T1118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 491990 T1184 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@346e83ca name:ZooKeeperConnection Watcher:127.0.0.1:44305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 491991 T1118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 491995 T1118 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 492999 T1118 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44330_
[junit4] 2> 493000 T1118 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44330_
[junit4] 2> 493003 T1118 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 493010 T1186 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 493010 T1186 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 493011 T1145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 493011 T1186 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 493014 T1141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "node_name":"127.0.0.1:44330_",
[junit4] 2> "roles":null,
[junit4] 2> "numShards":"2",
[junit4] 2> "base_url":"http://127.0.0.1:44330",
[junit4] 2> "state":"down"}
[junit4] 2> 493014 T1141 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 493014 T1141 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 493016 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 493017 T1145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 493017 T1166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 493017 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 494012 T1186 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 494013 T1186 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 494014 T1186 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 494014 T1186 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 494015 T1186 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 494015 T1186 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-004/collection1/'
[junit4] 2> 494018 T1186 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 494019 T1186 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 494020 T1186 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 494098 T1186 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 494168 T1186 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 494269 T1186 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 494279 T1186 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 494659 T1186 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 494662 T1186 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 494678 T1186 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 494684 T1186 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 494702 T1186 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 494702 T1186 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 494703 T1186 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-004/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty2/
[junit4] 2> 494703 T1186 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ac846a2
[junit4] 2> 494705 T1186 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty2
[junit4] 2> 494706 T1186 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty2/index/
[junit4] 2> 494707 T1186 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 494708 T1186 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty2/index
[junit4] 2> 494709 T1186 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=32, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 494742 T1186 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 494743 T1186 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 494749 T1186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 494749 T1186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 494749 T1186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 494750 T1186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 494750 T1186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 494750 T1186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 494751 T1186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 494751 T1186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 494751 T1186 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 494752 T1186 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 494752 T1186 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 494753 T1186 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 494754 T1186 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 494754 T1186 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 494754 T1186 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 494755 T1186 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 494755 T1186 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 494756 T1186 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 494756 T1186 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 494757 T1186 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 494783 T1186 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 494786 T1186 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 494788 T1186 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 494790 T1186 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 494794 T1186 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 494794 T1186 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 494795 T1186 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21149206554905187]
[junit4] 2> 494797 T1186 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 494797 T1186 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 494797 T1186 oass.SolrIndexSearcher.<init> Opening Searcher@77ec0243[collection1] main
[junit4] 2> 494798 T1186 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 494799 T1186 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 494799 T1186 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 494799 T1186 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 494800 T1186 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 494800 T1186 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 494801 T1186 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 494801 T1186 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 494801 T1186 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 494801 T1186 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 494805 T1187 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@77ec0243[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 494809 T1186 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 494810 T1190 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44330 collection:collection1 shard:shard1
[junit4] 2> 494810 T1118 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4] 2> 494810 T1118 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 494811 T1190 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 494815 T1190 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 494816 T1145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 494816 T1190 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 494817 T1190 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C308 name=collection1 org.apache.solr.core.SolrCore@7e083c7f url=http://127.0.0.1:44330/collection1 node=127.0.0.1:44330_ C308_STATE=coll:collection1 core:collection1 props:{core=collection1, node_name=127.0.0.1:44330_, base_url=http://127.0.0.1:44330, state=down}
[junit4] 2> 494817 T1190 C308 P44330 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44330/collection1/
[junit4] 2> 494817 T1190 C308 P44330 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 494817 T1190 C308 P44330 oasc.SyncStrategy.syncToMe http://127.0.0.1:44330/collection1/ has no replicas
[junit4] 2> 494817 T1190 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44330/collection1/ shard1
[junit4] 2> 494818 T1190 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 494821 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 494821 T1145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 494821 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 494821 T1166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 494827 T1145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 494828 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 494828 T1145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 494828 T1166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 494828 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 494872 T1190 oasc.ZkController.register We are http://127.0.0.1:44330/collection1/ and leader is http://127.0.0.1:44330/collection1/
[junit4] 2> 494873 T1190 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44330
[junit4] 2> 494873 T1190 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 494873 T1190 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 494873 T1190 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 494874 T1145 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 494877 T1141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "node_name":"127.0.0.1:44330_",
[junit4] 2> "roles":null,
[junit4] 2> "numShards":"2",
[junit4] 2> "base_url":"http://127.0.0.1:44330",
[junit4] 2> "state":"active"}
[junit4] 2> 494981 T1166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 494981 T1145 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 494981 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 494981 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 495892 T1118 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 495893 T1118 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 495895 T1118 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:44342
[junit4] 2> 495895 T1118 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 495896 T1118 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 495896 T1118 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-005
[junit4] 2> 495896 T1118 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-005/'
[junit4] 2> 495930 T1118 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-005/solr.xml
[junit4] 2> 495990 T1118 oasc.CoreContainer.<init> New CoreContainer 1165805839
[junit4] 2> 495991 T1118 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-005/]
[junit4] 2> 495992 T1118 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 495992 T1118 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 495992 T1118 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 495992 T1118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 495993 T1118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 495993 T1118 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 495993 T1118 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 495993 T1118 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 495994 T1118 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 495994 T1118 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 495994 T1118 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 495995 T1118 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 495995 T1118 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 495995 T1118 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 495995 T1118 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44305/solr
[junit4] 2> 495996 T1118 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 495996 T1118 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 495996 T1118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 495998 T1201 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3741b6d name:ZooKeeperConnection Watcher:127.0.0.1:44305 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 495998 T1118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 495999 T1118 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 496000 T1118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 496001 T1203 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4091df70 name:ZooKeeperConnection Watcher:127.0.0.1:44305/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[juni
[...truncated too long message...]
action_logs_total_size=0,transaction_logs_total_number=0}
[junit4] 2> 561054 T1118 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 561054 T1118 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 561054 T1230 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "node_name":"127.0.0.1:44330_",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:44330",
[junit4] 2> "state":"down"}
[junit4] 2> 561054 T1118 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 561055 T1118 C316 P44330 oasu.DirectUpdateHandler2.closeWriter Committing on IndexWriter close.
[junit4] 2> 561057 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> C316_STATE=coll:collection1 core:collection1 props:{core=collection1, node_name=127.0.0.1:44330_, base_url=http://127.0.0.1:44330, state=down, leader=true}
[junit4] 2> 561205 T1118 C316 P44330 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty2/index,segFN=segments_2,generation=2}
[junit4] 2> 561206 T1118 C316 P44330 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4] 2> 561210 T1118 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 561210 T1118 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 561210 T1118 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty2 [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty2;done=false>>]
[junit4] 2> 561210 T1118 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty2
[junit4] 2> 561211 T1118 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty2/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty2/index;done=false>>]
[junit4] 2> 561211 T1118 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001/tempDir-001/jetty2/index
[junit4] 2> 561211 T1118 oasc.Overseer.close Overseer (id=92654515837468680-127.0.0.1:44330_-n_0000000002) closing
[junit4] 2> 561212 T1230 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:44330_
[junit4] 2> 562713 T1185 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 562715 T1118 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4] 2> 566904 T1118 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 44342
[junit4] 2> 566905 T1118 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4] 2> 566907 T1118 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44305 44305
[junit4] 2> 568707 T1119 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44305 44305
[junit4] 2> 568708 T1119 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/aliases.json
[junit4] 2> 5 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/live_nodes
[junit4] 2> 3 /solr/overseer/queue
[junit4] 2> 3 /solr/overseer/collection-queue-work
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=40325B5E2A1DE510 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=en_CA -Dtests.timezone=Pacific/Saipan -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 86.0s J1 | ChaosMonkeySafeLeaderTest.testDistribSearch <<<
[junit4] > Throwable #1: java.lang.AssertionError: expected:<0> but was:<1>
[junit4] > at __randomizedtesting.SeedInfo.seed([40325B5E2A1DE510:C1D4D5465D42852C]:0)
[junit4] > at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:153)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 568722 T1118 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-40325B5E2A1DE510-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene50): {rnd_b=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), a_i=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), _version_=PostingsFormat(name=Direct), a_t=PostingsFormat(name=Direct)}, docValues:{}, sim=DefaultSimilarity, locale=en_CA, timezone=Pacific/Saipan
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=47679208,total=244842496
[junit4] 2> NOTE: All tests run in this JVM: [TestManagedResource, SharedFSAutoReplicaFailoverUtilsTest, ShowFileRequestHandlerTest, TestGroupingSearch, PathHierarchyTokenizerFactoryTest, PeerSyncTest, TestSchemaVersionResource, TestLMDirichletSimilarityFactory, SolrRequestParserTest, TestPartialUpdateDeduplication, TestFieldSortValues, TestFieldCollectionResource, DirectUpdateHandlerTest, PrimitiveFieldTypeTest, ExternalFileFieldSortTest, OpenCloseCoreStressTest, TestTolerantSearch, TestBM25SimilarityFactory, SpatialFilterTest, TestCloudPivotFacet, ZkControllerTest, ExitableDirectoryReaderTest, UpdateParamsTest, DistribCursorPagingTest, SortByFunctionTest, TestRecovery, TestLFUCache, DistributedSuggestComponentTest, DebugComponentTest, SolrCoreTest, PreAnalyzedUpdateProcessorTest, SolrIndexConfigTest, StressHdfsTest, PreAnalyzedFieldTest, TestZkChroot, RequiredFieldsTest, NotRequiredUniqueKeyTest, JSONWriterTest, SampleTest, TestDocumentBuilder, TestHighFrequencyDictionaryFactory, HighlighterMaxOffsetTest, SolrCoreCheckLockOnStartupTest, DocumentBuilderTest, TestAtomicUpdateErrorCases, AnalyticsQueryTest, URLClassifyProcessorTest, RankQueryTest, AnalyticsMergeStrategyTest, ChaosMonkeySafeLeaderTest]
[junit4] Completed on J1 in 86.01s, 1 test, 1 failure <<< FAILURES!
[...truncated 1063 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:524: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:961: There were test failures: 437 suites, 1795 tests, 1 failure, 58 ignored (22 assumptions)
Total time: 80 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-Tests-trunk-Java7 #4924
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 164171960 bytes
Compression is 0.0%
Took 30 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4927 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4927/
1 tests failed.
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
expected:<0> but was:<1>
Stack Trace:
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([693B4C7455CE853A:E8DDC26C2291E506]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:153)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11240 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/init-core-data-001
[junit4] 2> 689101 T555 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 689101 T555 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 689110 T555 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 689111 T555 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 689112 T556 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 689211 T555 oasc.ZkTestServer.run start zk server on port:28156
[junit4] 2> 689212 T555 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 689213 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 689218 T562 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4cf64ef8 name:ZooKeeperConnection Watcher:127.0.0.1:28156 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 689218 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 689218 T555 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 689219 T555 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 689222 T555 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 689223 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 689224 T564 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@680a7f84 name:ZooKeeperConnection Watcher:127.0.0.1:28156/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 689225 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 689225 T555 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 689225 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 689228 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 689230 T555 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 689231 T555 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 689233 T555 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 689234 T555 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 689237 T555 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 689238 T555 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 689340 T555 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 689341 T555 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 689343 T555 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 689344 T555 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 689346 T555 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 689347 T555 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 689349 T555 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 689350 T555 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 689352 T555 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 689353 T555 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 689355 T555 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 689356 T555 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 689358 T555 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 689358 T555 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 689361 T555 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 689361 T555 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 689363 T555 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 689364 T555 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 691052 T555 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 691056 T555 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:28159
[junit4] 2> 691057 T555 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 691057 T555 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 691058 T555 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-002
[junit4] 2> 691058 T555 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-002/'
[junit4] 2> 691101 T555 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-002/solr.xml
[junit4] 2> 691182 T555 oasc.CoreContainer.<init> New CoreContainer 1513081910
[junit4] 2> 691183 T555 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-002/]
[junit4] 2> 691184 T555 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 691184 T555 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 691184 T555 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 691185 T555 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 691185 T555 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 691185 T555 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 691186 T555 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 691186 T555 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 691186 T555 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 691186 T555 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 691187 T555 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 691187 T555 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 691188 T555 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 691188 T555 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 691188 T555 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:28156/solr
[junit4] 2> 691188 T555 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 691189 T555 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 691190 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 691191 T575 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78081b7b name:ZooKeeperConnection Watcher:127.0.0.1:28156 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 691191 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 691192 T555 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 691194 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 691195 T577 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6eb4d17a name:ZooKeeperConnection Watcher:127.0.0.1:28156/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 691195 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 691197 T555 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 691199 T555 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 691200 T555 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 691202 T555 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 691204 T555 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 691206 T555 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 691207 T555 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:28159_
[junit4] 2> 691208 T555 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:28159_
[junit4] 2> 691209 T555 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 691211 T555 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 691212 T555 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 691213 T555 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:28159_
[junit4] 2> 691214 T555 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 691215 T555 oasc.Overseer.start Overseer (id=92647856665395203-127.0.0.1:28159_-n_0000000000) starting
[junit4] 2> 691217 T555 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 691222 T555 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 691223 T579 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 691224 T555 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 691225 T555 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 691226 T555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 691228 T578 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 691243 T581 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 691243 T581 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 691245 T581 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 691245 T582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 691247 T578 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1:28159",
[junit4] 2> "roles":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "node_name":"127.0.0.1:28159_",
[junit4] 2> "state":"down"}
[junit4] 2> 691247 T578 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 691247 T578 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
[junit4] 2> 691247 T578 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 691249 T582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 692245 T581 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 692246 T581 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 692247 T581 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 692247 T581 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 692248 T581 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 692248 T581 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-002/collection1/'
[junit4] 2> 692250 T581 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 692250 T581 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 692251 T581 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 692303 T581 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 692341 T581 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 692442 T581 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 692461 T581 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 692861 T581 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 692864 T581 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 692866 T581 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 692872 T581 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 692908 T581 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 692909 T581 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 692991 T581 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/control/data/
[junit4] 2> 692991 T581 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@41a87e88
[junit4] 2> 692995 T581 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/control/data
[junit4] 2> 692996 T581 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/control/data/index/
[junit4] 2> 692997 T581 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 692998 T581 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/control/data/index
[junit4] 2> 692999 T581 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=15, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6426921365247941]
[junit4] 2> 693004 T581 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 693004 T581 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 693010 T581 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 693011 T581 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 693011 T581 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 693011 T581 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 693012 T581 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 693012 T581 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 693013 T581 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 693013 T581 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 693013 T581 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 693014 T581 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 693015 T581 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 693015 T581 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 693016 T581 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 693017 T581 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 693017 T581 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 693017 T581 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 693018 T581 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 693018 T581 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 693019 T581 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 693020 T581 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 693040 T581 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 693043 T581 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 693045 T581 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 693047 T581 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 693051 T581 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 693052 T581 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 693053 T581 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=64.3583984375, floorSegmentMB=1.7216796875, forceMergeDeletesPctAllowed=8.435338699319107, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5970489509632869
[junit4] 2> 693055 T581 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 693056 T581 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 693056 T581 oass.SolrIndexSearcher.<init> Opening Searcher@7a5e2929[collection1] main
[junit4] 2> 693056 T581 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 693058 T581 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 693058 T581 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 693058 T581 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 693059 T581 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 693059 T581 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 693060 T581 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 693060 T581 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 693061 T581 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
[junit4] 2> 693063 T581 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
[junit4] 2> 693063 T581 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 693064 T581 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 693067 T583 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7a5e2929[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 693071 T581 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 693072 T586 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:28159 collection:control_collection shard:shard1
[junit4] 2> 693073 T555 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4] 2> 693074 T555 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 693074 T586 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 693075 T555 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 693076 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 693078 T588 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5df758a7 name:ZooKeeperConnection Watcher:127.0.0.1:28156/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 693078 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 693078 T555 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 693080 T555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 693080 T586 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 693082 T582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 693083 T586 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 693083 T586 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 693083 T555 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> ASYNC NEW_CORE C42 name=collection1 org.apache.solr.core.SolrCore@5f910d86 url=http://127.0.0.1:28159/collection1 node=127.0.0.1:28159_ C42_STATE=coll:control_collection core:collection1 props:{base_url=http://127.0.0.1:28159, core=collection1, node_name=127.0.0.1:28159_, state=down}
[junit4] 2> 693083 T586 C42 P28159 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:28159/collection1/
[junit4] 2> 693084 T586 C42 P28159 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 693084 T586 C42 P28159 oasc.SyncStrategy.syncToMe http://127.0.0.1:28159/collection1/ has no replicas
[junit4] 2> 693085 T586 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:28159/collection1/ shard1
[junit4] 2> 693085 T586 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 693087 T582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 693087 T589 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 693097 T582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 693098 T589 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 693098 T582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 693141 T586 oasc.ZkController.register We are http://127.0.0.1:28159/collection1/ and leader is http://127.0.0.1:28159/collection1/
[junit4] 2> 693142 T586 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:28159
[junit4] 2> 693142 T586 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 693142 T586 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 693143 T586 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 693145 T582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 693148 T578 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1:28159",
[junit4] 2> "roles":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "node_name":"127.0.0.1:28159_",
[junit4] 2> "state":"active"}
[junit4] 2> 693251 T582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 693251 T589 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 694793 T555 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 694794 T555 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 694797 T555 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:28163
[junit4] 2> 694798 T555 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 694799 T555 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 694799 T555 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-003
[junit4] 2> 694800 T555 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-003/'
[junit4] 2> 694862 T555 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-003/solr.xml
[junit4] 2> 694975 T555 oasc.CoreContainer.<init> New CoreContainer 780346929
[junit4] 2> 694976 T555 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-003/]
[junit4] 2> 694977 T555 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 694978 T555 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 694978 T555 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 694978 T555 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 694979 T555 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 694979 T555 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 694980 T555 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 694980 T555 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 694981 T555 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 694981 T555 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 694982 T555 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 694982 T555 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 694983 T555 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 694983 T555 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 694984 T555 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:28156/solr
[junit4] 2> 694984 T555 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 694985 T555 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 694986 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 694987 T600 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@244072d2 name:ZooKeeperConnection Watcher:127.0.0.1:28156 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 694988 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 694989 T555 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 694991 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 694993 T602 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5cf7dba1 name:ZooKeeperConnection Watcher:127.0.0.1:28156/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 694993 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 694998 T555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 696002 T555 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:28163_
[junit4] 2> 696003 T555 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:28163_
[junit4] 2> 696007 T555 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 696014 T604 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 696015 T604 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 696016 T604 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 696016 T582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 696018 T578 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1:28163",
[junit4] 2> "roles":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "node_name":"127.0.0.1:28163_",
[junit4] 2> "state":"down"}
[junit4] 2> 696019 T578 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 696019 T578 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 1
[junit4] 2> 696019 T578 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 696022 T603 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 696022 T589 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 696022 T582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 697016 T604 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 697017 T604 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 697018 T604 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 697018 T604 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 697019 T604 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 697019 T604 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-003/collection1/'
[junit4] 2> 697021 T604 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 697022 T604 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 697022 T604 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 697093 T604 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 697146 T604 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 697247 T604 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 697269 T604 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 697655 T604 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 697658 T604 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 697660 T604 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 697665 T604 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 697697 T604 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 697698 T604 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 697698 T604 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-003/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty1/
[junit4] 2> 697699 T604 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@41a87e88
[junit4] 2> 697700 T604 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty1
[junit4] 2> 697701 T604 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty1/index/
[junit4] 2> 697702 T604 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 697704 T604 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty1/index
[junit4] 2> 697704 T604 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=15, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6426921365247941]
[junit4] 2> 697709 T604 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 697709 T604 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 697716 T604 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 697717 T604 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 697717 T604 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 697718 T604 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 697718 T604 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 697718 T604 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 697719 T604 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 697719 T604 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 697720 T604 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 697721 T604 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 697721 T604 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 697722 T604 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 697723 T604 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 697723 T604 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 697724 T604 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 697724 T604 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 697725 T604 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 697725 T604 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 697727 T604 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 697728 T604 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 697746 T604 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 697748 T604 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 697750 T604 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 697752 T604 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 697756 T604 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 697756 T604 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 697758 T604 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=64.3583984375, floorSegmentMB=1.7216796875, forceMergeDeletesPctAllowed=8.435338699319107, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5970489509632869
[junit4] 2> 697759 T604 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 697760 T604 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 697760 T604 oass.SolrIndexSearcher.<init> Opening Searcher@7867574[collection1] main
[junit4] 2> 697761 T604 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 697762 T604 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 697762 T604 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 697763 T604 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 697763 T604 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 697763 T604 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 697764 T604 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 697765 T604 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 697765 T604 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 697765 T604 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 697770 T605 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7867574[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 697775 T604 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 697776 T608 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:28163 collection:collection1 shard:shard1
[junit4] 2> 697776 T555 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4] 2> 697777 T555 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 697777 T608 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 697782 T608 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 697784 T582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 697784 T608 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 697784 T608 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C43 name=collection1 org.apache.solr.core.SolrCore@218fbbbd url=http://127.0.0.1:28163/collection1 node=127.0.0.1:28163_ C43_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:28163, core=collection1, node_name=127.0.0.1:28163_, state=down}
[junit4] 2> 697784 T608 C43 P28163 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:28163/collection1/
[junit4] 2> 697785 T608 C43 P28163 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 697785 T608 C43 P28163 oasc.SyncStrategy.syncToMe http://127.0.0.1:28163/collection1/ has no replicas
[junit4] 2> 697785 T608 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:28163/collection1/ shard1
[junit4] 2> 697786 T608 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 697788 T589 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 697788 T582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 697788 T603 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 697794 T582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 697795 T582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 697795 T589 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 697795 T603 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 697841 T608 oasc.ZkController.register We are http://127.0.0.1:28163/collection1/ and leader is http://127.0.0.1:28163/collection1/
[junit4] 2> 697842 T608 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:28163
[junit4] 2> 697842 T608 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 697842 T608 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 697842 T608 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 697844 T582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 697847 T578 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1:28163",
[junit4] 2> "roles":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "node_name":"127.0.0.1:28163_",
[junit4] 2> "state":"active"}
[junit4] 2> 697950 T582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 697950 T603 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 697950 T589 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 698976 T555 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 698977 T555 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 698979 T555 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:28170
[junit4] 2> 698980 T555 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 698980 T555 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 698981 T555 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-004
[junit4] 2> 698981 T555 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-004/'
[junit4] 2> 699011 T555 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-004/solr.xml
[junit4] 2> 699068 T555 oasc.CoreContainer.<init> New CoreContainer 2075921453
[junit4] 2> 699068 T555 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-004/]
[junit4] 2> 699070 T555 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 699070 T555 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 699070 T555 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 699070 T555 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 699071 T555 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 699071 T555 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 699071 T555 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 699072 T555 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 699072 T555 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 699072 T555 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 699073 T555 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 699073 T555 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 699073 T555 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 699074 T555 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 699074 T555 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:28156/solr
[junit4] 2> 699074 T555 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 699074 T555 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 699075 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 699077 T619 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c3e2537 name:ZooKeeperConnection Watcher:127.0.0.1:28156 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 699078 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 699079 T555 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 699082 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 699083 T621 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@496e7927 name:ZooKeeperConnection Watcher:127.0.0.1:28156/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 699083 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 699088 T555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 700091 T555 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:28170_
[junit4] 2> 700092 T555 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:28170_
[junit4] 2> 700096 T555 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 700105 T623 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 700105 T623 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 700106 T582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 700106 T623 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 700109 T578 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1:28170",
[junit4] 2> "roles":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "node_name":"127.0.0.1:28170_",
[junit4] 2> "state":"down"}
[junit4] 2> 700109 T578 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4] 2> 700110 T578 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 700112 T582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 700112 T589 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 700112 T622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 700112 T603 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 701107 T623 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 701108 T623 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 701109 T623 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 701109 T623 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 701110 T623 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 701110 T623 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-004/collection1/'
[junit4] 2> 701112 T623 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 701113 T623 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 701114 T623 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 701200 T623 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 701255 T623 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 701357 T623 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 701373 T623 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 701784 T623 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 701788 T623 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 701790 T623 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 701795 T623 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 701827 T623 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 701827 T623 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 701828 T623 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-004/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty2/
[junit4] 2> 701828 T623 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@41a87e88
[junit4] 2> 701830 T623 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty2
[junit4] 2> 701831 T623 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty2/index/
[junit4] 2> 701832 T623 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 701834 T623 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty2/index
[junit4] 2> 701834 T623 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=15, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6426921365247941]
[junit4] 2> 701909 T623 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 701909 T623 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 701917 T623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 701917 T623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 701918 T623 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 701918 T623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 701918 T623 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 701919 T623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 701919 T623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 701920 T623 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 701920 T623 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 701921 T623 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 701922 T623 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 701922 T623 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 701923 T623 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 701924 T623 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 701924 T623 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 701925 T623 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 701925 T623 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 701926 T623 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 701927 T623 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 701927 T623 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 701946 T623 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 701948 T623 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 701950 T623 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 701952 T623 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 701956 T623 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 701957 T623 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 701958 T623 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=64.3583984375, floorSegmentMB=1.7216796875, forceMergeDeletesPctAllowed=8.435338699319107, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5970489509632869
[junit4] 2> 701960 T623 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 701961 T623 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 701961 T623 oass.SolrIndexSearcher.<init> Opening Searcher@685f44ec[collection1] main
[junit4] 2> 701962 T623 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 701963 T623 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 701963 T623 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 701963 T623 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 701964 T623 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 701964 T623 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 701965 T623 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 701966 T623 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 701966 T623 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 701966 T623 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 701971 T624 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@685f44ec[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 701976 T623 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 701977 T627 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:28170 collection:collection1 shard:shard1
[junit4] 2> 701977 T555 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4] 2> 701978 T555 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 701980 T627 oasc.ZkController.register We are http://127.0.0.1:28170/collection1/ and leader is http://127.0.0.1:28163/collection1/
[junit4] 2> 701980 T627 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:28170
[junit4] 2> 701980 T627 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 701981 T555 oasc.AbstractFullDistribZkTestBase.enableAutoSoftCommit Turning on auto soft commit: 1000
[junit4] 2> 701981 T627 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C44 name=collection1 org.apache.solr.core.SolrCore@33d1fc75 url=http://127.0.0.1:28170/collection1 node=127.0.0.1:28170_ C44_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:28170, core=collection1, node_name=127.0.0.1:28170_, state=down}
[junit4] 2> 701982 T628 C44 P28170 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 701982 T628 C44 P28170 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 701983 T628 C44 P28170 oasc.RecoveryStrategy.doRecovery Publishing state of core collection1 as recovering, leader is http://127.0.0.1:28163/collection1/ and I am http://127.0.0.1:28170/collection1/
[junit4] 2> 701983 T628 C44 P28170 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
[junit4] 2> 701984 T628 C44 P28170 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 701986 T582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 701987 T628 C44 P28170 oasc.RecoveryStrategy.sendPrepRecoveryCmd Sending prep recovery command to http://127.0.0.1:28163; WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1%3A28170_&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
[junit4] 2> ASYNC NEW_CORE C45 name=collection1 org.apache.solr.core.SolrCore@5f910d86 url=http://127.0.0.1:28159/collection1 node=127.0.0.1:28159_ C45_STATE=coll:control_collection core:collection1 props:{base_url=http://127.0.0.1:28159, core=collection1, node_name=127.0.0.1:28159_, state=active, leader=true}
[junit4] 2> 701989 T567 C45 P28159 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {deleteByQuery=*:* (-1482365720137302016)} 0 5
[junit4] 2> 701989 T593 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 701990 T578 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1:28170",
[junit4] 2> "roles":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "node_name":"127.0.0.1:28170_",
[junit4] 2> "state":"recovering"}
[junit4] 2> 701992 T593 oasha.CoreAdminHandler.handleWaitForStateAction Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
[junit4] 2> 701993 T593 oasha.CoreAdminHandler.handleWaitForStateAction In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:28170_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"base_url":"http://127.0.0.1:28170","core":"collection1","node_name":"127.0.0.1:28170_","state":"down"}
[junit4] 2> 701994 T622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 701994 T582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 701994 T603 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 701994 T589 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> C44_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:28170, core=collection1, node_name=127.0.0.1:28170_, state=recovering}
[junit4] 2> 702002 T611 C44 P28170 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&_version_=-1482365720146739200&distrib.from=http://127.0.0.1:28163/collection1/} {deleteByQuery=*:* (-1482365720146739200)} 0 3
[junit4] 2> ASYNC NEW_CORE C46 name=collection1 org.apache.solr.core.SolrCore@218fbbbd url=http://127.0.0.1:28163/collection1 node=127.0.0.1:28163_ C46_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:28163, core=collection1, node_name=127.0.0.1:28163_, state=active, leader=true}
[junit4] 2> 702003 T594 C46 P28163 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {deleteByQuery=*:* (-1482365720146739200)} 0 11
[junit4] 2> 702009 T555 oasc.ChaosMonkey.monkeyLog monkey: starting
[junit4] 2> 702010 T555 oasc.ChaosMonkey.monkeyLog monkey: Jetty will not commit on close
[junit4] 2> 702061 T567 C45 P28159 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-0 (1482365720168759296)]} 0 48
[junit4] 2> 702061 T570 C45 P28159 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[0-0 (1482365720167710720)]} 0 49
[junit4] 2> 702079 T612 C44 P28170 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:28163/collection1/} {add=[1-0 (1482365720222236673)]} 0 6
[junit4] 2> 702079 T611 C44 P28170 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:28163/collection1/} {add=[0-0 (1482365720222236672)]} 0 6
[junit4] 2> 702080 T594 C46 P28163 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[1-0 (1482365720222236673)]} 0 15
[junit4] 2> 702080 T595 C46 P28163 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[0-0 (1482365720222236672)]} 0 16
[junit4] 2> 702140 T570 C45 P28159 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-1 (1482365720299831297)]} 0 2
[junit4] 2> 702140 T567 C45 P28159 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[0-1 (1482365720299831296)]} 0 2
[junit4] 2> 702150 T612 C44 P28170 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:28163/collection1/} {add=[1-1 (1482365720305074177)]} 0 2
[junit4] 2> 702150 T611 C44 P28170 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:28163/collection1/} {add=[0-1 (1482365720305074176)]} 0 2
[junit4] 2> 702150 T595 C46 P28163 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[1-1 (1482365720305074177)]} 0 6
[junit4] 2> 702151 T594 C46 P28163 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[0-1 (1482365720305074176)]} 0 7
[junit4] 2> 702187 T567 C45 P28159 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[0-2 (1482365720349114369)]} 0 2
[junit4] 2> 702187 T570 C45 P28159 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-2 (1482365720349114368)]} 0 2
[junit4] 2> 702197 T612 C44 P28170 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:28163/collection1/} {add=[0-2 (1482365720354357249)]} 0 2
[junit4] 2> 702197 T611 C44 P28170 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:28163/collection1/} {add=[1-2 (1482365720354357248)]} 0 2
[junit4] 2> 702198 T594 C46 P28163 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[0-2 (1482365720354357249)]} 0 7
[junit4] 2> 702198 T595 C46 P28163 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[1-2 (1482365720354357248)]} 0 7
[junit4] 2> 702210 T570 C45 P28159 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[0-1 (-1482365720373231617)]} 0 2
[junit4] 2> 702210 T567 C45 P28159 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[1-1 (-1482365720373231616)]} 0 2
[junit4] 2> 702217 T611 C44 P28170 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:28163/collection1/} {delete=[1-1 (-1482365720378474497)]} 0 1
[junit4] 2> 702217 T612 C44 P28170 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:28163/collection1/} {delete=[0-1 (-1482365720378474496)]} 0 1
[junit4] 2> 702218 T594 C46 P28163 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[1-1 (-1482365720378474497)]} 0 5
[junit4] 2> 702218 T595 C46 P28163 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {delete=[0-1 (-1482365720378474496)]} 0 5
[junit4] 2> 702223 T567 C45 P28159 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={
[...truncated too long message...]
{/,null}
[junit4] 2> 756506 T555 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 28163
[junit4] 2> 756507 T555 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=780346929
[junit4] 2> 756508 T555 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 756508 T555 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 756509 T642 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 756510 T555 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@218fbbbd
[junit4] 2> 756513 T643 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1:28163",
[junit4] 2> "roles":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "node_name":"127.0.0.1:28163_",
[junit4] 2> "state":"down"}
[junit4] 2> 756515 T555 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=7,autocommits=0,soft autocommit maxTime=1000ms,soft autocommits=6,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=37,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=125,cumulative_deletesById=55,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=9735,transaction_logs_total_number=1}
[junit4] 2> 756515 T555 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 756515 T555 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 756516 T555 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 756515 T642 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 756516 T555 C49 P28163 oasu.DirectUpdateHandler2.closeWriter Committing on IndexWriter close.
[junit4] 2> C49_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:28163, core=collection1, node_name=127.0.0.1:28163_, state=down, leader=true}
[junit4] 2> 756558 T555 C49 P28163 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty1/index,segFN=segments_2,generation=2}
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty1/index,segFN=segments_3,generation=3}
[junit4] 2> 756559 T555 C49 P28163 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4] 2> 756567 T555 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 756570 T555 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 756570 T555 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty1 [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty1;done=false>>]
[junit4] 2> 756571 T555 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty1
[junit4] 2> 756571 T555 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty1/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty1/index;done=false>>]
[junit4] 2> 756571 T555 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001/tempDir-001/jetty1/index
[junit4] 2> 756572 T555 oasc.Overseer.close Overseer (id=92647856665395206-127.0.0.1:28163_-n_0000000001) closing
[junit4] 2> 756573 T643 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:28163_
[junit4] 2> 758074 T642 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 758076 T555 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4] 2> 761708 T555 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 28170
[junit4] 2> 761709 T555 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4] 2> 761711 T555 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:28156 28156
[junit4] 2> 762875 T556 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:28156 28156
[junit4] 2> 762877 T556 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/clusterstate.json
[junit4] 2> 4 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/live_nodes
[junit4] 2> 2 /solr/overseer/queue
[junit4] 2> 2 /solr/overseer/collection-queue-work
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=693B4C7455CE853A -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=ko -Dtests.timezone=AGT -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 73.8s J1 | ChaosMonkeySafeLeaderTest.testDistribSearch <<<
[junit4] > Throwable #1: java.lang.AssertionError: expected:<0> but was:<1>
[junit4] > at __randomizedtesting.SeedInfo.seed([693B4C7455CE853A:E8DDC26C2291E506]:0)
[junit4] > at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:153)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 762945 T555 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-693B4C7455CE853A-001
[junit4] 2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=55), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=55)), sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=ko, timezone=AGT
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=110560656,total=240123904
[junit4] 2> NOTE: All tests run in this JVM: [TestManagedResource, TestDocumentBuilder, HttpPartitionTest, ClusterStateTest, SystemInfoHandlerTest, JSONWriterTest, AnalyticsMergeStrategyTest, DocumentBuilderTest, TestCollapseQParserPlugin, SolrRequestParserTest, ChaosMonkeySafeLeaderTest]
[junit4] Completed on J1 in 73.90s, 1 test, 1 failure <<< FAILURES!
[...truncated 1019 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:524: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:961: There were test failures: 437 suites, 1794 tests, 1 failure, 58 ignored (22 assumptions)
Total time: 82 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-Tests-trunk-Java7 #4924
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 164068886 bytes
Compression is 0.0%
Took 29 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4926 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4926/
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
expected:<0> but was:<1>
Stack Trace:
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([BD924E273F298EF0:3C74C03F4876EECC]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:153)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11222 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/init-core-data-001
[junit4] 2> 587238 T1217 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
[junit4] 2> 587238 T1217 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 587246 T1217 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 587247 T1217 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 587248 T1218 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 587348 T1217 oasc.ZkTestServer.run start zk server on port:59949
[junit4] 2> 587349 T1217 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 587349 T1217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 587353 T1224 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17a4087b name:ZooKeeperConnection Watcher:127.0.0.1:59949 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 587353 T1217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 587354 T1217 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 587354 T1217 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 587357 T1217 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 587358 T1217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 587359 T1226 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@310c2bc0 name:ZooKeeperConnection Watcher:127.0.0.1:59949/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 587359 T1217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 587359 T1217 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 587360 T1217 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 587362 T1217 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 587364 T1217 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 587366 T1217 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 587368 T1217 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 587369 T1217 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 587372 T1217 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 587373 T1217 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 587476 T1217 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 587477 T1217 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 587481 T1217 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 587482 T1217 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 587486 T1217 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 587487 T1217 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 587491 T1217 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 587491 T1217 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 587494 T1217 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 587494 T1217 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 587497 T1217 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 587497 T1217 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 587500 T1217 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 587500 T1217 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 587503 T1217 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 587503 T1217 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 587506 T1217 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 587506 T1217 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 587510 T1217 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 587510 T1217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 587512 T1228 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6dc56837 name:ZooKeeperConnection Watcher:127.0.0.1:59949/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 587512 T1217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 587512 T1217 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 588658 T1217 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 588664 T1217 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 588669 T1217 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:59955
[junit4] 2> 588671 T1217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 588671 T1217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 588671 T1217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-002
[junit4] 2> 588672 T1217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-002/'
[junit4] 2> 588702 T1217 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-002/solr.xml
[junit4] 2> 588757 T1217 oasc.CoreContainer.<init> New CoreContainer 479387567
[junit4] 2> 588757 T1217 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-002/]
[junit4] 2> 588758 T1217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 588758 T1217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 588758 T1217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 588759 T1217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 588759 T1217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 588759 T1217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 588759 T1217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 588760 T1217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 588760 T1217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 588760 T1217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 588762 T1217 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 588764 T1217 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 588764 T1217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 588764 T1217 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 588765 T1217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59949/solr
[junit4] 2> 588765 T1217 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 588765 T1217 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 588766 T1217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 588768 T1239 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c33cfe3 name:ZooKeeperConnection Watcher:127.0.0.1:59949 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 588768 T1217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 588769 T1217 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 588771 T1217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 588772 T1241 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29c56f37 name:ZooKeeperConnection Watcher:127.0.0.1:59949/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 588772 T1217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 588774 T1217 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 588777 T1217 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 588780 T1217 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 588783 T1217 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 588786 T1217 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 588790 T1217 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 588791 T1217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59955_
[junit4] 2> 588792 T1217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59955_
[junit4] 2> 588795 T1217 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 588797 T1217 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 588799 T1217 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 588801 T1217 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:59955_
[junit4] 2> 588802 T1217 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 588804 T1217 oasc.Overseer.start Overseer (id=92643741301735428-127.0.0.1:59955_-n_0000000000) starting
[junit4] 2> 588806 T1217 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 588815 T1217 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 588816 T1243 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 588817 T1217 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 588820 T1217 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 588821 T1217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 588825 T1242 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 588831 T1245 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 588831 T1245 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 588832 T1245 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 588832 T1246 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 588835 T1242 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "base_url":"https://127.0.0.1:59955",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":null,
[junit4] 2> "state":"down",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "operation":"state",
[junit4] 2> "node_name":"127.0.0.1:59955_",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1"}
[junit4] 2> 588835 T1242 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 588836 T1242 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
[junit4] 2> 588836 T1242 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 588839 T1246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 589832 T1245 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 589833 T1245 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 589834 T1245 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 589834 T1245 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 589835 T1245 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 589835 T1245 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-002/collection1/'
[junit4] 2> 589837 T1245 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 589838 T1245 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 589838 T1245 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 589903 T1245 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 589951 T1245 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 590053 T1245 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 590063 T1245 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 590416 T1245 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 590419 T1245 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 590420 T1245 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 590425 T1245 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 590448 T1245 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 590448 T1245 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 590514 T1245 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/control/data/
[junit4] 2> 590514 T1245 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@15802f1
[junit4] 2> 590517 T1245 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/control/data
[junit4] 2> 590518 T1245 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/control/data/index/
[junit4] 2> 590519 T1245 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 590520 T1245 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/control/data/index
[junit4] 2> 590520 T1245 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=19, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8097127315077258]
[junit4] 2> 590525 T1245 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 590526 T1245 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 590530 T1245 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 590531 T1245 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 590531 T1245 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 590531 T1245 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 590532 T1245 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 590532 T1245 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 590532 T1245 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 590533 T1245 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 590533 T1245 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 590534 T1245 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 590534 T1245 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 590534 T1245 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 590535 T1245 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 590535 T1245 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 590536 T1245 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 590536 T1245 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 590536 T1245 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 590537 T1245 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 590537 T1245 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 590538 T1245 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 590556 T1245 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 590558 T1245 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 590560 T1245 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 590562 T1245 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 590565 T1245 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 590565 T1245 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 590566 T1245 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 590568 T1245 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 590568 T1245 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 590568 T1245 oass.SolrIndexSearcher.<init> Opening Searcher@6a2dd3bc[collection1] main
[junit4] 2> 590569 T1245 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 590570 T1245 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 590570 T1245 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 590570 T1245 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 590571 T1245 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 590571 T1245 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 590572 T1245 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 590572 T1245 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 590573 T1245 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
[junit4] 2> 590575 T1245 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
[junit4] 2> 590575 T1245 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 590576 T1245 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 590579 T1247 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6a2dd3bc[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 590582 T1245 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 590582 T1250 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:59955 collection:control_collection shard:shard1
[junit4] 2> 590583 T1217 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4] 2> 590583 T1217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 590584 T1250 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 590589 T1217 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 590590 T1250 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 590590 T1217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 590591 T1252 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44fd5538 name:ZooKeeperConnection Watcher:127.0.0.1:59949/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 590592 T1217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 590592 T1217 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 590592 T1250 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 590593 T1250 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 590593 T1246 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C256 name=collection1 org.apache.solr.core.SolrCore@72259ea9 url=https://127.0.0.1:59955/collection1 node=127.0.0.1:59955_ C256_STATE=coll:control_collection core:collection1 props:{base_url=https://127.0.0.1:59955, state=down, node_name=127.0.0.1:59955_, core=collection1}
[junit4] 2> 590593 T1250 C256 P59955 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:59955/collection1/
[junit4] 2> 590594 T1217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 590594 T1250 C256 P59955 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 590594 T1250 C256 P59955 oasc.SyncStrategy.syncToMe https://127.0.0.1:59955/collection1/ has no replicas
[junit4] 2> 590594 T1250 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:59955/collection1/ shard1
[junit4] 2> 590595 T1250 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 590598 T1217 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 590600 T1246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 590600 T1253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 590608 T1246 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 590609 T1246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 590609 T1253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 590652 T1250 oasc.ZkController.register We are https://127.0.0.1:59955/collection1/ and leader is https://127.0.0.1:59955/collection1/
[junit4] 2> 590653 T1250 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:59955
[junit4] 2> 590653 T1250 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 590653 T1250 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 590653 T1250 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 590655 T1246 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 590659 T1242 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "base_url":"https://127.0.0.1:59955",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "state":"active",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "operation":"state",
[junit4] 2> "node_name":"127.0.0.1:59955_",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1"}
[junit4] 2> 590763 T1253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 590763 T1246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 592165 T1217 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 592166 T1217 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 592172 T1217 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 592175 T1217 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:59981
[junit4] 2> 592178 T1217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 592178 T1217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 592178 T1217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-003
[junit4] 2> 592179 T1217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-003/'
[junit4] 2> 592219 T1217 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-003/solr.xml
[junit4] 2> 592306 T1217 oasc.CoreContainer.<init> New CoreContainer 1036753793
[junit4] 2> 592306 T1217 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-003/]
[junit4] 2> 592307 T1217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 592308 T1217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 592308 T1217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 592308 T1217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 592309 T1217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 592309 T1217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 592309 T1217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 592310 T1217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 592310 T1217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 592310 T1217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 592314 T1217 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 592316 T1217 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 592317 T1217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 592317 T1217 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 592317 T1217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59949/solr
[junit4] 2> 592318 T1217 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 592318 T1217 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 592319 T1217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 592321 T1264 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@162b59a8 name:ZooKeeperConnection Watcher:127.0.0.1:59949 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 592321 T1217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 592322 T1217 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 592324 T1217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 592325 T1266 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6076dcd4 name:ZooKeeperConnection Watcher:127.0.0.1:59949/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 592325 T1217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 592333 T1217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 593338 T1217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59981_
[junit4] 2> 593339 T1217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59981_
[junit4] 2> 593344 T1217 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 593351 T1268 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 593351 T1268 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 593352 T1246 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 593352 T1268 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 593356 T1242 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "base_url":"https://127.0.0.1:59981",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "state":"down",
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state",
[junit4] 2> "node_name":"127.0.0.1:59981_",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1"}
[junit4] 2> 593356 T1242 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 593356 T1242 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 1
[junit4] 2> 593356 T1242 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 593359 T1253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 593359 T1267 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 593359 T1246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 594353 T1268 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 594354 T1268 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 594355 T1268 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 594355 T1268 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 594356 T1268 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 594356 T1268 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-003/collection1/'
[junit4] 2> 594358 T1268 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 594359 T1268 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 594359 T1268 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 594421 T1268 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 594466 T1268 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 594567 T1268 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 594576 T1268 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 594910 T1268 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 594913 T1268 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 594914 T1268 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 594919 T1268 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 594944 T1268 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 594945 T1268 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 594945 T1268 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-003/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty1/
[junit4] 2> 594945 T1268 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@15802f1
[junit4] 2> 594947 T1268 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty1
[junit4] 2> 594948 T1268 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty1/index/
[junit4] 2> 594948 T1268 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 594950 T1268 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty1/index
[junit4] 2> 594950 T1268 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=19, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8097127315077258]
[junit4] 2> 594954 T1268 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 594955 T1268 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 594961 T1268 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 594961 T1268 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 594962 T1268 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 594962 T1268 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 594962 T1268 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 594962 T1268 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 594963 T1268 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 594963 T1268 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 594963 T1268 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 594964 T1268 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 594965 T1268 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 594965 T1268 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 594966 T1268 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 594967 T1268 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 594967 T1268 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 594967 T1268 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 594968 T1268 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 594968 T1268 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 594969 T1268 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 594969 T1268 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 594988 T1268 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 594989 T1268 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 594991 T1268 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 594993 T1268 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 594997 T1268 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 594997 T1268 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 594998 T1268 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 595000 T1268 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 595000 T1268 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 595000 T1268 oass.SolrIndexSearcher.<init> Opening Searcher@8cdbfe2[collection1] main
[junit4] 2> 595000 T1268 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 595002 T1268 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 595002 T1268 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 595002 T1268 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 595003 T1268 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 595003 T1268 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 595004 T1268 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 595004 T1268 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 595004 T1268 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 595005 T1268 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 595009 T1269 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@8cdbfe2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 595014 T1268 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 595014 T1272 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:59981 collection:collection1 shard:shard1
[junit4] 2> 595015 T1217 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4] 2> 595015 T1217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 595015 T1272 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 595022 T1272 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 595024 T1246 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 595024 T1272 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 595024 T1272 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C257 name=collection1 org.apache.solr.core.SolrCore@d776863 url=https://127.0.0.1:59981/collection1 node=127.0.0.1:59981_ C257_STATE=coll:collection1 core:collection1 props:{base_url=https://127.0.0.1:59981, state=down, node_name=127.0.0.1:59981_, core=collection1}
[junit4] 2> 595025 T1272 C257 P59981 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:59981/collection1/
[junit4] 2> 595025 T1272 C257 P59981 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 595025 T1272 C257 P59981 oasc.SyncStrategy.syncToMe https://127.0.0.1:59981/collection1/ has no replicas
[junit4] 2> 595025 T1272 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:59981/collection1/ shard1
[junit4] 2> 595025 T1272 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 595030 T1267 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 595030 T1253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 595030 T1246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 595038 T1246 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 595039 T1267 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 595039 T1246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 595039 T1253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 595082 T1272 oasc.ZkController.register We are https://127.0.0.1:59981/collection1/ and leader is https://127.0.0.1:59981/collection1/
[junit4] 2> 595083 T1272 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:59981
[junit4] 2> 595083 T1272 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 595083 T1272 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 595083 T1272 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 595085 T1246 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 595088 T1242 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "base_url":"https://127.0.0.1:59981",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "state":"active",
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state",
[junit4] 2> "node_name":"127.0.0.1:59981_",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1"}
[junit4] 2> 595192 T1267 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 595192 T1246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 595192 T1253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 596456 T1217 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 596457 T1217 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 596463 T1217 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 596466 T1217 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:59988
[junit4] 2> 596470 T1217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 596470 T1217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 596470 T1217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-004
[junit4] 2> 596471 T1217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-004/'
[junit4] 2> 596515 T1217 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-004/solr.xml
[junit4] 2> 596591 T1217 oasc.CoreContainer.<init> New CoreContainer 85217603
[junit4] 2> 596592 T1217 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-004/]
[junit4] 2> 596593 T1217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 596593 T1217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 596593 T1217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 596593 T1217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 596594 T1217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 596594 T1217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 596594 T1217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 596594 T1217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 596594 T1217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 596595 T1217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 596597 T1217 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 596599 T1217 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 596599 T1217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 596599 T1217 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 596599 T1217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59949/solr
[junit4] 2> 596600 T1217 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 596600 T1217 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 596601 T1217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 596602 T1283 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76efed40 name:ZooKeeperConnection Watcher:127.0.0.1:59949 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 596603 T1217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 596603 T1217 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 596606 T1217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 596607 T1285 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7cf96b07 name:ZooKeeperConnection Watcher:127.0.0.1:59949/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 596607 T1217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 596614 T1217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 597619 T1217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59988_
[junit4] 2> 597620 T1217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59988_
[junit4] 2> 597625 T1217 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 597633 T1287 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 597633 T1287 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 597634 T1246 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 597634 T1287 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 597637 T1242 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "base_url":"https://127.0.0.1:59988",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "state":"down",
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state",
[junit4] 2> "node_name":"127.0.0.1:59988_",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1"}
[junit4] 2> 597638 T1242 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 597638 T1242 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 597641 T1253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 597641 T1246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 597641 T1286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 597641 T1267 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 598635 T1287 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 598636 T1287 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 598637 T1287 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 598637 T1287 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 598638 T1287 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 598638 T1287 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-004/collection1/'
[junit4] 2> 598641 T1287 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 598642 T1287 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 598643 T1287 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 598710 T1287 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 598766 T1287 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 598868 T1287 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 598878 T1287 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 599238 T1287 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 599240 T1287 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 599242 T1287 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 599247 T1287 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 599270 T1287 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 599270 T1287 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 599271 T1287 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-004/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty2/
[junit4] 2> 599271 T1287 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@15802f1
[junit4] 2> 599273 T1287 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty2
[junit4] 2> 599273 T1287 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty2/index/
[junit4] 2> 599274 T1287 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 599275 T1287 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty2/index
[junit4] 2> 599276 T1287 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=19, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8097127315077258]
[junit4] 2> 599288 T1287 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 599289 T1287 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 599295 T1287 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 599295 T1287 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 599296 T1287 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 599296 T1287 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 599296 T1287 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 599296 T1287 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 599297 T1287 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 599297 T1287 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 599297 T1287 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 599298 T1287 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 599299 T1287 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 599299 T1287 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 599300 T1287 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 599300 T1287 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 599301 T1287 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 599301 T1287 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 599302 T1287 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 599302 T1287 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 599303 T1287 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 599303 T1287 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 599321 T1287 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 599323 T1287 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 599325 T1287 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 599327 T1287 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 599330 T1287 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 599331 T1287 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 599332 T1287 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 599333 T1287 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 599333 T1287 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 599334 T1287 oass.SolrIndexSearcher.<init> Opening Searcher@58e32eb3[collection1] main
[junit4] 2> 599334 T1287 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 599335 T1287 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 599335 T1287 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 599336 T1287 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 599336 T1287 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 599336 T1287 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 599337 T1287 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 599338 T1287 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 599338 T1287 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 599338 T1287 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 599343 T1288 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@58e32eb3[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 599347 T1287 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 599348 T1291 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:59988 collection:collection1 shard:shard2
[junit4] 2> 599348 T1217 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
[junit4] 2> 599349 T1217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 599349 T1291 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 599355 T1291 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 599357 T1246 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 599357 T1291 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 599357 T1291 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C258 name=collection1 org.apache.solr.core.SolrCore@2da8e183 url=https://127.0.0.1:59988/collection1 node=127.0.0.1:59988_ C258_STATE=coll:collection1 core:collection1 props:{base_url=https://127.0.0.1:59988, state=down, node_name=127.0.0.1:59988_, core=collection1}
[junit4] 2> 599357 T1291 C258 P59988 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:59988/collection1/
[junit4] 2> 599358 T1291 C258 P59988 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 599358 T1291 C258 P59988 oasc.SyncStrategy.syncToMe https://127.0.0.1:59988/collection1/ has no replicas
[junit4] 2> 599358 T1291 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:59988/collection1/ shard2
[junit4] 2> 599358 T1291 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 599363 T1246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 599363 T1267 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 599363 T1286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 599364 T1253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 599372 T1246 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 599373 T1246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 599373 T1253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 599373 T1267 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 599373 T1286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 599414 T1291 oasc.ZkController.register We are https://127.0.0.1:59988/collection1/ and leader is https://127.0.0.1:59988/collection1/
[junit4] 2> 599415 T1291 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:59988
[junit4] 2> 599415 T1291 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 599415 T1291 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 599415 T1291 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 599417 T1246 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 599420 T1242 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "base_url":"https://127.0.0.1:59988",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard2",
[junit4] 2> "state":"active",
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state",
[junit4] 2> "node_name":"127.0.0.1:59988_",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1"}
[junit4] 2> 599524 T1267 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 599524 T1286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 599524 T1246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 599524 T1253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 600372 T1217 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 600372 T1217 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 600377 T1217 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 600379 T1217 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:60011
[junit4] 2> 600381 T1217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 600382 T1217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 600382 T1217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-005
[junit4] 2> 600382 T1217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-005/'
[junit4] 2> 600411 T1217 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-005/solr.xml
[junit4] 2> 600464 T1217 oasc.CoreContainer.<init> New CoreContainer 1372798401
[junit4] 2> 600464 T1217 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-005/]
[junit4] 2> 600465 T1217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 600465 T1217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 600465 T1217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 600465 T1217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 600466 T1217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 600466 T1217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 600466 T1217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 600466 T1217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 600467 T1217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 600467 T1217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 600469 T1217 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 600471 T1217 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 600471 T1217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 600471 T1217 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 600471 T1217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59949/solr
[junit4] 2> 600472 T1217
[...truncated too long message...]
Handler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=34,adds=34,deletesById=19,deletesByQuery=1,errors=0,cumulative_adds=34,cumulative_deletesById=19,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
[junit4] 2> 661600 T1217 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 661600 T1217 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 661600 T1217 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 661601 T1217 C268 P59988 oasu.DirectUpdateHandler2.closeWriter Committing on IndexWriter close.
[junit4] 2> 661601 T1763 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "base_url":"https://127.0.0.1:59988",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "shard":"shard2",
[junit4] 2> "state":"down",
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state",
[junit4] 2> "node_name":"127.0.0.1:59988_",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1"}
[junit4] 2> 661604 T1762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> C268_STATE=coll:collection1 core:collection1 props:{base_url=https://127.0.0.1:59988, state=down, node_name=127.0.0.1:59988_, core=collection1, leader=true}
[junit4] 2> 661670 T1217 C268 P59988 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty2/index,segFN=segments_2,generation=2}
[junit4] 2> 661671 T1217 C268 P59988 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4] 2> 661674 T1217 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 661675 T1217 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 661675 T1217 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty2 [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty2;done=false>>]
[junit4] 2> 661675 T1217 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty2
[junit4] 2> 661676 T1217 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty2/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty2/index;done=false>>]
[junit4] 2> 661676 T1217 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001/tempDir-001/jetty2/index
[junit4] 2> 661676 T1217 oasc.Overseer.close Overseer (id=92643741301735433-127.0.0.1:59988_-n_0000000002) closing
[junit4] 2> 661677 T1763 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:59988_
[junit4] 2> 663179 T1762 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 663181 T1217 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4] 2> 666733 T1217 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 60011
[junit4] 2> 666734 T1217 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4] 2> 666735 T1217 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:59949 59949
[junit4] 2> 666942 T1218 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:59949 59949
[junit4] 2> 666943 T1218 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/clusterstate.json
[junit4] 2> 5 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/live_nodes
[junit4] 2> 3 /solr/overseer/collection-queue-work
[junit4] 2> 3 /solr/overseer/queue
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=BD924E273F298EF0 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=fr_LU -Dtests.timezone=America/Bogota -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 79.7s J1 | ChaosMonkeySafeLeaderTest.testDistribSearch <<<
[junit4] > Throwable #1: java.lang.AssertionError: expected:<0> but was:<1>
[junit4] > at __randomizedtesting.SeedInfo.seed([BD924E273F298EF0:3C74C03F4876EECC]:0)
[junit4] > at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:153)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 666967 T1217 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BD924E273F298EF0-001
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=fr_LU, timezone=America/Bogota
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=188246960,total=292552704
[junit4] 2> NOTE: All tests run in this JVM: [TestCollationField, SharedFSAutoReplicaFailoverUtilsTest, TestFastOutputStream, SpatialFilterTest, SharedFSAutoReplicaFailoverTest, PreAnalyzedFieldTest, UpdateParamsTest, DebugComponentTest, URLClassifyProcessorTest, HighlighterMaxOffsetTest, TestConfig, TestCollationFieldDocValues, TestBulkSchemaAPI, TestCollapseQParserPlugin, PeerSyncTest, TestBM25SimilarityFactory, NotRequiredUniqueKeyTest, CurrencyFieldOpenExchangeTest, TestReload, TestManagedSchemaFieldTypeResource, SolrCoreTest, TestClassNameShortening, SuggesterFSTTest, VMParamsZkACLAndCredentialsProvidersTest, ClusterStateTest, ZkControllerTest, SolrCoreCheckLockOnStartupTest, TestManagedResource, SystemInfoHandlerTest, MBeansHandlerTest, TestAddFieldRealTimeGet, AddBlockUpdateTest, DocumentBuilderTest, DistributedSuggestComponentTest, TestQuerySenderNoQuery, TestFunctionQuery, ShowFileRequestHandlerTest, AnalyticsMergeStrategyTest, PreAnalyzedUpdateProcessorTest, TestRecovery, TestManagedSchemaFieldResource, DistribCursorPagingTest, SortByFunctionTest, TestAtomicUpdateErrorCases, TestIntervalFaceting, UUIDUpdateProcessorFallbackTest, TestDocumentBuilder, TestFieldSortValues, TestLFUCache, TestSolrXmlPersistor, CloudExitableDirectoryReaderTest, ChaosMonkeySafeLeaderTest]
[junit4] Completed on J1 in 79.77s, 1 test, 1 failure <<< FAILURES!
[...truncated 1032 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:524: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:961: There were test failures: 437 suites, 1794 tests, 1 failure, 58 ignored (22 assumptions)
Total time: 77 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-Tests-trunk-Java7 #4924
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 160222191 bytes
Compression is 0.0%
Took 28 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure