You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2017/02/19 09:55:21 UTC
[JENKINS-EA] Lucene-Solr-6.x-Linux (32bit/jdk-9-ea+155) - Build #
2891 - Still Failing!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/2891/
Java: 32bit/jdk-9-ea+155 -client -XX:+UseParallelGC
All tests passed
Build Log:
[...truncated 51629 lines...]
check-lib-versions:
[echo] Lib versions check under: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/..
[libversions] :: loading settings :: file = /home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/top-level-ivy-settings.xml
[libversions] ORPHAN coordinate key '/cglib/cglib-nodep' in ivy-versions.properties is not found in any ivy.xml file.
[libversions] Checked that ivy-versions.properties and ivy-ignore-conflicts.properties have lexically sorted '/org/name' keys and no duplicates or orphans.
[libversions] Scanned 49 ivy.xml files for rev="${/org/name}" format.
[libversions] Found 0 indirect dependency version conflicts.
[libversions] Completed in 1.06s., 1 error(s).
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:775: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:122: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/build.xml:108: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/tools/custom-tasks.xml:108: Lib versions check failed. Check the logs.
Total time: 72 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-6.x-Linux (64bit/jdk1.8.0_121) - Build # 2894
- Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/2894/
Java: 64bit/jdk1.8.0_121 -XX:+UseCompressedOops -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test
Error Message:
Expected 2 of 3 replicas to be active but only found 1; [core_node2:{"core":"c8n_1x3_lf_shard1_replica3","base_url":"http://127.0.0.1:38214","node_name":"127.0.0.1:38214_","state":"active","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/16)={ "replicationFactor":"3", "shards":{"shard1":{ "range":"80000000-7fffffff", "state":"active", "replicas":{ "core_node1":{ "state":"down", "base_url":"http://127.0.0.1:38070", "core":"c8n_1x3_lf_shard1_replica1", "node_name":"127.0.0.1:38070_"}, "core_node2":{ "core":"c8n_1x3_lf_shard1_replica3", "base_url":"http://127.0.0.1:38214", "node_name":"127.0.0.1:38214_", "state":"active", "leader":"true"}, "core_node3":{ "core":"c8n_1x3_lf_shard1_replica2", "base_url":"http://127.0.0.1:45871", "node_name":"127.0.0.1:45871_", "state":"down"}}}}, "router":{"name":"compositeId"}, "maxShardsPerNode":"1", "autoAddReplicas":"false"}
Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node2:{"core":"c8n_1x3_lf_shard1_replica3","base_url":"http://127.0.0.1:38214","node_name":"127.0.0.1:38214_","state":"active","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/16)={
"replicationFactor":"3",
"shards":{"shard1":{
"range":"80000000-7fffffff",
"state":"active",
"replicas":{
"core_node1":{
"state":"down",
"base_url":"http://127.0.0.1:38070",
"core":"c8n_1x3_lf_shard1_replica1",
"node_name":"127.0.0.1:38070_"},
"core_node2":{
"core":"c8n_1x3_lf_shard1_replica3",
"base_url":"http://127.0.0.1:38214",
"node_name":"127.0.0.1:38214_",
"state":"active",
"leader":"true"},
"core_node3":{
"core":"c8n_1x3_lf_shard1_replica2",
"base_url":"http://127.0.0.1:45871",
"node_name":"127.0.0.1:45871_",
"state":"down"}}}},
"router":{"name":"compositeId"},
"maxShardsPerNode":"1",
"autoAddReplicas":"false"}
at __randomizedtesting.SeedInfo.seed([7634F286EDC1E95:8F3770F2C020736D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:168)
at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11267 lines...]
[junit4] Suite: org.apache.solr.cloud.LeaderFailoverAfterPartitionTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/init-core-data-001
[junit4] 2> 88689 INFO (SUITE-LeaderFailoverAfterPartitionTest-seed#[7634F286EDC1E95]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields
[junit4] 2> 88691 INFO (SUITE-LeaderFailoverAfterPartitionTest-seed#[7634F286EDC1E95]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
[junit4] 2> 88691 INFO (SUITE-LeaderFailoverAfterPartitionTest-seed#[7634F286EDC1E95]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
[junit4] 2> 88693 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 88693 INFO (Thread-187) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 88693 INFO (Thread-187) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 88793 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.ZkTestServer start zk server on port:34150
[junit4] 2> 88802 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 88803 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 88804 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 88804 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 88805 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 88806 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 88807 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 88808 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 88808 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 88809 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 88810 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 88873 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/control-001/cores/collection1
[junit4] 2> 88875 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 88875 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@27e41906{/,null,AVAILABLE}
[junit4] 2> 88877 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@3685f9f9{HTTP/1.1,[http/1.1]}{127.0.0.1:35898}
[junit4] 2> 88877 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.Server Started @91491ms
[junit4] 2> 88877 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/tempDir-001/control/data, hostContext=/, hostPort=38214, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/control-001/cores}
[junit4] 2> 88878 ERROR (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 88878 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.5.0
[junit4] 2> 88878 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 88878 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 88878 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-19T17:02:03.335Z
[junit4] 2> 88890 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 88890 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/control-001/solr.xml
[junit4] 2> 88899 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 88899 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34150/solr
[junit4] 2> 88914 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:38214_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 88915 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:38214_ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:38214_
[junit4] 2> 88915 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:38214_ ] o.a.s.c.Overseer Overseer (id=97486354720423940-127.0.0.1:38214_-n_0000000000) starting
[junit4] 2> 88921 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:38214_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38214_
[junit4] 2> 88922 INFO (zkCallback-90-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 89019 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:38214_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/control-001/cores
[junit4] 2> 89019 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:38214_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 89023 INFO (OverseerStateUpdate-97486354720423940-127.0.0.1:38214_-n_0000000000) [n:127.0.0.1:38214_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 90036 WARN (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 90037 INFO (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
[junit4] 2> 90069 INFO (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 90209 WARN (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 90211 INFO (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 90230 INFO (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 90231 INFO (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/control-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/control-001/cores/collection1/data/]
[junit4] 2> 90231 INFO (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@24586e0b
[junit4] 2> 90233 INFO (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=99.70703125, floorSegmentMB=1.2197265625, forceMergeDeletesPctAllowed=29.597210789439046, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.12272637107496741
[junit4] 2> 90251 WARN (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 90303 INFO (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 90303 INFO (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 90304 INFO (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 90304 INFO (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 90304 INFO (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=25, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.16139120002134488]
[junit4] 2> 90305 INFO (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@30ef80b1[collection1] main]
[junit4] 2> 90306 INFO (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 90306 INFO (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 90307 INFO (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 90307 INFO (searcherExecutor-465-thread-1-processing-n:127.0.0.1:38214_ x:collection1 c:control_collection) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@30ef80b1[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 90308 INFO (coreLoadExecutor-464-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ c:control_collection x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559781677219184640
[junit4] 2> 90314 INFO (coreZkRegister-457-thread-1-processing-n:127.0.0.1:38214_ x:collection1 c:control_collection) [n:127.0.0.1:38214_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 90314 INFO (coreZkRegister-457-thread-1-processing-n:127.0.0.1:38214_ x:collection1 c:control_collection) [n:127.0.0.1:38214_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 90314 INFO (coreZkRegister-457-thread-1-processing-n:127.0.0.1:38214_ x:collection1 c:control_collection) [n:127.0.0.1:38214_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:38214/collection1/
[junit4] 2> 90314 INFO (coreZkRegister-457-thread-1-processing-n:127.0.0.1:38214_ x:collection1 c:control_collection) [n:127.0.0.1:38214_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 90314 INFO (coreZkRegister-457-thread-1-processing-n:127.0.0.1:38214_ x:collection1 c:control_collection) [n:127.0.0.1:38214_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:38214/collection1/ has no replicas
[junit4] 2> 90317 INFO (coreZkRegister-457-thread-1-processing-n:127.0.0.1:38214_ x:collection1 c:control_collection) [n:127.0.0.1:38214_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:38214/collection1/ shard1
[junit4] 2> 90468 INFO (coreZkRegister-457-thread-1-processing-n:127.0.0.1:38214_ x:collection1 c:control_collection) [n:127.0.0.1:38214_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 90571 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 90572 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34150/solr ready
[junit4] 2> 90574 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 90574 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
[junit4] 2> 90678 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-1-001/cores/collection1
[junit4] 2> 90679 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-1-001
[junit4] 2> 90679 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 90680 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@697556b3{/,null,AVAILABLE}
[junit4] 2> 90681 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@264c734f{HTTP/1.1,[http/1.1]}{127.0.0.1:43308}
[junit4] 2> 90681 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.Server Started @93294ms
[junit4] 2> 90681 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=37992, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-1-001/cores}
[junit4] 2> 90682 ERROR (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 90682 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.5.0
[junit4] 2> 90682 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 90682 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 90682 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-19T17:02:05.139Z
[junit4] 2> 90685 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 90685 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-1-001/solr.xml
[junit4] 2> 90690 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 90690 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34150/solr
[junit4] 2> 90697 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:37992_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 90698 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:37992_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 90700 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:37992_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37992_
[junit4] 2> 90701 INFO (zkCallback-94-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 90701 INFO (zkCallback-90-thread-3-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 90701 INFO (zkCallback-100-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 90795 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:37992_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-1-001/cores
[junit4] 2> 90795 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:37992_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 90798 INFO (OverseerStateUpdate-97486354720423940-127.0.0.1:38214_-n_0000000000) [n:127.0.0.1:38214_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
[junit4] 2> 90900 INFO (zkCallback-100-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 91806 WARN (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 91806 INFO (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
[junit4] 2> 91818 INFO (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 91893 WARN (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 91896 INFO (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 91904 INFO (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 91904 INFO (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-1-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-1-001/cores/collection1/data/]
[junit4] 2> 91905 INFO (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@24586e0b
[junit4] 2> 91906 INFO (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=99.70703125, floorSegmentMB=1.2197265625, forceMergeDeletesPctAllowed=29.597210789439046, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.12272637107496741
[junit4] 2> 91911 WARN (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 91938 INFO (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 91939 INFO (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 91941 INFO (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 91941 INFO (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 91942 INFO (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=25, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.16139120002134488]
[junit4] 2> 91943 INFO (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2393b3a4[collection1] main]
[junit4] 2> 91944 INFO (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 91944 INFO (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 91944 INFO (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 91945 INFO (searcherExecutor-476-thread-1-processing-n:127.0.0.1:37992_ x:collection1 c:collection1) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2393b3a4[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 91945 INFO (coreLoadExecutor-475-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559781678935703552
[junit4] 2> 91950 INFO (coreZkRegister-470-thread-1-processing-n:127.0.0.1:37992_ x:collection1 c:collection1) [n:127.0.0.1:37992_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 91950 INFO (coreZkRegister-470-thread-1-processing-n:127.0.0.1:37992_ x:collection1 c:collection1) [n:127.0.0.1:37992_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 91950 INFO (coreZkRegister-470-thread-1-processing-n:127.0.0.1:37992_ x:collection1 c:collection1) [n:127.0.0.1:37992_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:37992/collection1/
[junit4] 2> 91950 INFO (coreZkRegister-470-thread-1-processing-n:127.0.0.1:37992_ x:collection1 c:collection1) [n:127.0.0.1:37992_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 91950 INFO (coreZkRegister-470-thread-1-processing-n:127.0.0.1:37992_ x:collection1 c:collection1) [n:127.0.0.1:37992_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:37992/collection1/ has no replicas
[junit4] 2> 91954 INFO (coreZkRegister-470-thread-1-processing-n:127.0.0.1:37992_ x:collection1 c:collection1) [n:127.0.0.1:37992_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:37992/collection1/ shard2
[junit4] 2> 92064 INFO (zkCallback-100-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 92105 INFO (coreZkRegister-470-thread-1-processing-n:127.0.0.1:37992_ x:collection1 c:collection1) [n:127.0.0.1:37992_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 92208 INFO (zkCallback-100-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 92397 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-2-001/cores/collection1
[junit4] 2> 92398 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-2-001
[junit4] 2> 92399 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 92400 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6dd5c998{/,null,AVAILABLE}
[junit4] 2> 92400 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@2c9da68f{HTTP/1.1,[http/1.1]}{127.0.0.1:34721}
[junit4] 2> 92400 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.Server Started @95014ms
[junit4] 2> 92400 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=45871, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-2-001/cores}
[junit4] 2> 92401 ERROR (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 92401 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.5.0
[junit4] 2> 92401 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 92401 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 92401 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-19T17:02:06.858Z
[junit4] 2> 92404 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 92404 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-2-001/solr.xml
[junit4] 2> 92410 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 92411 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34150/solr
[junit4] 2> 92418 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:45871_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 92419 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:45871_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 92420 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:45871_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45871_
[junit4] 2> 92421 INFO (zkCallback-100-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 92421 INFO (zkCallback-106-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 92421 INFO (zkCallback-94-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 92421 INFO (zkCallback-90-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 92524 INFO (zkCallback-100-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 92556 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:45871_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-2-001/cores
[junit4] 2> 92556 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:45871_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 92558 INFO (OverseerStateUpdate-97486354720423940-127.0.0.1:38214_-n_0000000000) [n:127.0.0.1:38214_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 92659 INFO (zkCallback-100-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 92659 INFO (zkCallback-106-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 93569 WARN (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 93570 INFO (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
[junit4] 2> 93589 INFO (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 93665 WARN (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 93667 INFO (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 93676 INFO (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 93676 INFO (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-2-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-2-001/cores/collection1/data/]
[junit4] 2> 93676 INFO (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@24586e0b
[junit4] 2> 93677 INFO (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=99.70703125, floorSegmentMB=1.2197265625, forceMergeDeletesPctAllowed=29.597210789439046, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.12272637107496741
[junit4] 2> 93682 WARN (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 93709 INFO (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 93709 INFO (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 93710 INFO (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 93710 INFO (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 93711 INFO (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=25, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.16139120002134488]
[junit4] 2> 93711 INFO (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@689e8da[collection1] main]
[junit4] 2> 93712 INFO (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 93712 INFO (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 93712 INFO (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 93714 INFO (searcherExecutor-487-thread-1-processing-n:127.0.0.1:45871_ x:collection1 c:collection1) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@689e8da[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 93714 INFO (coreLoadExecutor-486-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559781680790634496
[junit4] 2> 93718 INFO (coreZkRegister-481-thread-1-processing-n:127.0.0.1:45871_ x:collection1 c:collection1) [n:127.0.0.1:45871_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 93718 INFO (coreZkRegister-481-thread-1-processing-n:127.0.0.1:45871_ x:collection1 c:collection1) [n:127.0.0.1:45871_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 93718 INFO (coreZkRegister-481-thread-1-processing-n:127.0.0.1:45871_ x:collection1 c:collection1) [n:127.0.0.1:45871_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:45871/collection1/
[junit4] 2> 93718 INFO (coreZkRegister-481-thread-1-processing-n:127.0.0.1:45871_ x:collection1 c:collection1) [n:127.0.0.1:45871_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 93718 INFO (coreZkRegister-481-thread-1-processing-n:127.0.0.1:45871_ x:collection1 c:collection1) [n:127.0.0.1:45871_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:45871/collection1/ has no replicas
[junit4] 2> 93720 INFO (coreZkRegister-481-thread-1-processing-n:127.0.0.1:45871_ x:collection1 c:collection1) [n:127.0.0.1:45871_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:45871/collection1/ shard1
[junit4] 2> 93822 INFO (zkCallback-100-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 93822 INFO (zkCallback-106-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 93871 INFO (coreZkRegister-481-thread-1-processing-n:127.0.0.1:45871_ x:collection1 c:collection1) [n:127.0.0.1:45871_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 93974 INFO (zkCallback-100-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 93974 INFO (zkCallback-106-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 94160 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-3-001/cores/collection1
[junit4] 2> 94160 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-3-001
[junit4] 2> 94161 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 94163 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@48282eac{/,null,AVAILABLE}
[junit4] 2> 94164 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@5e0fa31b{HTTP/1.1,[http/1.1]}{127.0.0.1:39962}
[junit4] 2> 94164 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.Server Started @96777ms
[junit4] 2> 94164 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/, hostPort=38070, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-3-001/cores}
[junit4] 2> 94164 ERROR (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 94164 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.5.0
[junit4] 2> 94164 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 94164 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 94164 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-19T17:02:08.621Z
[junit4] 2> 94168 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 94168 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-3-001/solr.xml
[junit4] 2> 94173 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 94173 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34150/solr
[junit4] 2> 94182 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:38070_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 94183 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:38070_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 94184 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:38070_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38070_
[junit4] 2> 94185 INFO (zkCallback-100-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 94185 INFO (zkCallback-90-thread-3-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 94185 INFO (zkCallback-94-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 94185 INFO (zkCallback-112-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 94185 INFO (zkCallback-106-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 94270 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:38070_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-3-001/cores
[junit4] 2> 94270 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [n:127.0.0.1:38070_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 94272 INFO (OverseerStateUpdate-97486354720423940-127.0.0.1:38214_-n_0000000000) [n:127.0.0.1:38214_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
[junit4] 2> 94374 INFO (zkCallback-100-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 94374 INFO (zkCallback-112-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 94374 INFO (zkCallback-106-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 95281 WARN (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 95282 INFO (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
[junit4] 2> 95297 INFO (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 95362 WARN (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 95364 INFO (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 95372 INFO (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 95372 INFO (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-3-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001/shard-3-001/cores/collection1/data/]
[junit4] 2> 95373 INFO (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@24586e0b
[junit4] 2> 95374 INFO (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=99.70703125, floorSegmentMB=1.2197265625, forceMergeDeletesPctAllowed=29.597210789439046, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.12272637107496741
[junit4] 2> 95382 WARN (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 95419 INFO (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 95419 INFO (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 95420 INFO (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 95420 INFO (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 95421 INFO (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=25, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.16139120002134488]
[junit4] 2> 95421 INFO (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@73c770c0[collection1] main]
[junit4] 2> 95422 INFO (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 95422 INFO (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 95422 INFO (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 95424 INFO (searcherExecutor-498-thread-1-processing-n:127.0.0.1:38070_ x:collection1 c:collection1) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@73c770c0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 95424 INFO (coreLoadExecutor-497-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559781682583699456
[junit4] 2> 95427 INFO (coreZkRegister-492-thread-1-processing-n:127.0.0.1:38070_ x:collection1 c:collection1) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
[junit4] 2> 95427 INFO (updateExecutor-109-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 95427 INFO (recoveryExecutor-110-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 95428 INFO (recoveryExecutor-110-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 95428 INFO (recoveryExecutor-110-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
[junit4] 2> 95428 INFO (recoveryExecutor-110-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 95428 INFO (recoveryExecutor-110-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:37992/collection1/] and I am [http://127.0.0.1:38070/collection1/]
[junit4] 2> 95430 INFO (recoveryExecutor-110-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:37992]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:38070_&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 95431 INFO (SocketProxy-Acceptor-37992) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=50438,localport=37992], receiveBufferSize:531000
[junit4] 2> 95434 INFO (SocketProxy-Acceptor-37992) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43308,localport=39470], receiveBufferSize=530904
[junit4] 2> 95435 INFO (qtp2056456524-763) [n:127.0.0.1:37992_ ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 95435 INFO (qtp2056456524-763) [n:127.0.0.1:37992_ ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard2 of collection1) have state: recovering
[junit4] 2> 95436 INFO (qtp2056456524-763) [n:127.0.0.1:37992_ ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard2, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:38070_, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:38070","node_name":"127.0.0.1:38070_","state":"down"}
[junit4] 2> 95531 INFO (zkCallback-100-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 95531 INFO (zkCallback-112-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 95531 INFO (zkCallback-106-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 95810 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.SolrTestCaseJ4 ###Starting test
[junit4] 2> 95810 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Wait for recoveries to finish - wait 30000 for each attempt
[junit4] 2> 95810 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30000
[junit4] 2> 96436 INFO (qtp2056456524-763) [n:127.0.0.1:37992_ ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard2, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:38070_, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:38070","node_name":"127.0.0.1:38070_","state":"recovering"}
[junit4] 2> 96436 INFO (qtp2056456524-763) [n:127.0.0.1:37992_ ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 96436 INFO (qtp2056456524-763) [n:127.0.0.1:37992_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:38070_&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node3&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1001
[junit4] 2> 103437 INFO (recoveryExecutor-110-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:37992/collection1/] - recoveringAfterStartup=[true]
[junit4] 2> 103437 INFO (recoveryExecutor-110-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=http://127.0.0.1:38070 START replicas=[http://127.0.0.1:37992/collection1/] nUpdates=100
[junit4] 2> 103439 INFO (SocketProxy-Acceptor-37992) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=50458,localport=37992], receiveBufferSize:531000
[junit4] 2> 103440 INFO (SocketProxy-Acceptor-37992) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43308,localport=39490], receiveBufferSize=530904
[junit4] 2> 103445 INFO (qtp2056456524-763) [n:127.0.0.1:37992_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:2.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 103445 INFO (qtp2056456524-763) [n:127.0.0.1:37992_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=4
[junit4] 2> 103447 INFO (recoveryExecutor-110-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 103447 INFO (recoveryExecutor-110-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
[junit4] 2> 103447 INFO (recoveryExecutor-110-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 103447 INFO (recoveryExecutor-110-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 103448 INFO (recoveryExecutor-110-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 103448 INFO (recoveryExecutor-110-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
[junit4] 2> 103448 INFO (recoveryExecutor-110-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
[junit4] 2> 103448 INFO (recoveryExecutor-110-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
[junit4] 2> 103448 INFO (recoveryExecutor-110-thread-1-processing-n:127.0.0.1:38070_ x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
[junit4] 2> 103449 INFO (zkCallback-100-thread-1-processing-n:127.0.0.1:37992_) [n:127.0.0.1:37992_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 103449 INFO (zkCallback-112-thread-1-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 103449 INFO (zkCallback-106-thread-1-processing-n:127.0.0.1:45871_) [n:127.0.0.1:45871_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 103815 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
[junit4] 2> 103815 INFO (SocketProxy-Acceptor-38214) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=34068,localport=38214], receiveBufferSize:531000
[junit4] 2> 103817 INFO (SocketProxy-Acceptor-38214) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=35898,localport=43804], receiveBufferSize=530904
[junit4] 2> 103818 INFO (qtp804028863-724) [n:127.0.0.1:38214_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 103819 INFO (qtp804028863-724) [n:127.0.0.1:38214_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 103819 INFO (qtp804028863-724) [n:127.0.0.1:38214_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 103819 INFO (qtp804028863-724) [n:127.0.0.1:38214_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 1
[junit4] 2> 103822 INFO (SocketProxy-Acceptor-37992) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=50466,localport=37992], receiveBufferSize:531000
[junit4] 2> 103824 INFO (SocketProxy-Acceptor-37992) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43308,localport=39498], receiveBufferSize=530904
[junit4] 2> 103828 INFO (SocketProxy-Acceptor-45871) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=60480,localport=45871], receiveBufferSize:531000
[junit4] 2> 103828 INFO (SocketProxy-Acceptor-37992) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=50470,localport=37992], receiveBufferSize:531000
[junit4] 2> 103828 INFO (SocketProxy-Acceptor-38070) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=55326,localport=38070], receiveBufferSize:531000
[junit4] 2> 103828 INFO (SocketProxy-Acceptor-37992) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43308,localport=39506], receiveBufferSize=530904
[junit4] 2> 103829 INFO (SocketProxy-Acceptor-45871) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=34721,localport=53398], receiveBufferSize=530904
[junit4] 2> 103830 INFO (SocketProxy-Acceptor-38070) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=39962,localport=51178], receiveBufferSize=530904
[junit4] 2> 103831 INFO (qtp2056456524-763) [n:127.0.0.1:37992_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 103831 INFO (qtp2056456524-763) [n:127.0.0.1:37992_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 103831 INFO (qtp893249061-792) [n:127.0.0.1:45871_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 103833 INFO (qtp893249061-792) [n:127.0.0.1:45871_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 103834 INFO (qtp2056456524-763) [n:127.0.0.1:37992_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 103834 INFO (qtp893249061-792) [n:127.0.0.1:45871_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 103834 INFO (qtp2056456524-763) [n:127.0.0.1:37992_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:37992/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 3
[junit4] 2> 103834 INFO (qtp893249061-792) [n:127.0.0.1:45871_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:37992/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 3
[junit4] 2> 103835 INFO (qtp314229101-821) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 103836 INFO (qtp314229101-821) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 103836 INFO (qtp314229101-821) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 103836 INFO (qtp314229101-821) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:37992/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 103836 INFO (qtp2056456524-765) [n:127.0.0.1:37992_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={_stateVer_=collection1:10&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 11
[junit4] 2> 103838 INFO (SocketProxy-Acceptor-37992) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=50482,localport=37992], receiveBufferSize:531000
[junit4] 2> 103838 INFO (SocketProxy-Acceptor-37992) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43308,localport=39514], receiveBufferSize=530904
[junit4] 2> 103841 INFO (qtp2056456524-761) [n:127.0.0.1:37992_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=1
[junit4] 2> 103842 INFO (SocketProxy-Acceptor-38070) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=55334,localport=38070], receiveBufferSize:531000
[junit4] 2> 103843 INFO (SocketProxy-Acceptor-38070) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=39962,localport=51186], receiveBufferSize=530904
[junit4] 2> 103844 INFO (qtp314229101-822) [n:127.0.0.1:38070_ c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 103845 INFO (SocketProxy-Acceptor-45871) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=60498,localport=45871], receiveBufferSize:531000
[junit4] 2> 103846 INFO (SocketProxy-Acceptor-45871) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=34721,localport=53416], receiveBufferSize=530904
[junit4] 2> 103849 INFO (qtp893249061-794) [n:127.0.0.1:45871_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.S.Request [collection1] webapp= path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 105851 INFO (SocketProxy-Acceptor-37992) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=50502,localport=37992], receiveBufferSize:531000
[junit4] 2> 105852 INFO (SocketProxy-Acceptor-37992) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43308,localport=39534], receiveBufferSize=530904
[junit4] 2> 105853 INFO (qtp2056456524-765) [n:127.0.0.1:37992_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=3&maxShardsPerNode=1&name=c8n_1x3_lf&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 105855 INFO (OverseerThreadFactory-462-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ ] o.a.s.c.CreateCollectionCmd Create collection c8n_1x3_lf
[junit4] 2> 105855 INFO (OverseerThreadFactory-462-thread-1-processing-n:127.0.0.1:38214_) [n:127.0.0.1:38214_ ] o.a.s.c.CreateCollectionCmd Only one config set found in zk - using it:conf1
[junit4] 2> 105960 INFO (SocketProxy-Acceptor-38070) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=55354,localport=38070], receiveBufferSize:531000
[junit4] 2> 105960 INFO (SocketProxy-Acceptor-38214) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=34114,localport=38214], receiveBufferSize:531000
[junit4] 2> 105961 INFO (SocketProxy-Acceptor-45871) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=60522,localport=45871], receiveBufferSize:531000
[junit4] 2> 105961 INFO (SocketProxy-Acceptor-38214) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=35898,localport=43852], receiveBufferSize=530904
[junit4] 2> 105961 INFO (SocketProxy-Acceptor
[...truncated too long message...]
ation(ZkCmdExecutor.java:60)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:388)
[junit4] 2> at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:747)
[junit4] 2> at org.apache.solr.cloud.Overseer.getStateUpdateQueue(Overseer.java:620)
[junit4] 2> at org.apache.solr.cloud.Overseer.getStateUpdateQueue(Overseer.java:607)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:310)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:677)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:340)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:268)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 234257 INFO (zkCallback-112-thread-5-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:c8n_1x3_lf s:shard1 r:core_node1 x:c8n_1x3_lf_shard1_replica1] o.a.s.c.SolrCore [c8n_1x3_lf_shard1_replica1] CLOSING SolrCore org.apache.solr.core.SolrCore@19391ff7
[junit4] 2> 234257 WARN (zkCallback-112-thread-5-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:c8n_1x3_lf s:shard1 r:core_node1 x:c8n_1x3_lf_shard1_replica1] o.a.s.c.RecoveryStrategy Stopping recovery for core=[c8n_1x3_lf_shard1_replica1] coreNodeName=[core_node1]
[junit4] 2> 234280 INFO (zkCallback-112-thread-5-processing-n:127.0.0.1:38070_) [n:127.0.0.1:38070_ c:c8n_1x3_lf s:shard1 r:core_node1 x:c8n_1x3_lf_shard1_replica1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.c8n_1x3_lf.shard1.replica1
[junit4] 2> 234280 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 234281 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@5e0fa31b{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 234281 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@48282eac{/,null,UNAVAILABLE}
[junit4] 2> 234282 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34150 34150
[junit4] 2> 239302 INFO (Thread-187) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34150 34150
[junit4] 2> 239303 WARN (Thread-187) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/aliases.json
[junit4] 2> 5 /solr/clusterprops.json
[junit4] 2> 4 /solr/security.json
[junit4] 2> 4 /solr/configs/conf1
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/clusterstate.json
[junit4] 2> 3 /solr/collections/c8n_1x3_lf/state.json
[junit4] 2> 3 /solr/collections/collection1/state.json
[junit4] 2> 2 /solr/overseer_elect/election/97486354720423945-127.0.0.1:37992_-n_0000000001
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 141 /solr/overseer/collection-queue-work
[junit4] 2> 46 /solr/overseer/queue
[junit4] 2> 24 /solr/overseer/queue-work
[junit4] 2> 5 /solr/live_nodes
[junit4] 2> 5 /solr/collections
[junit4] 2>
[junit4] 2> 239303 WARN (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.SocketProxy Closing 1 connections to: http://127.0.0.1:38070/, target: http://127.0.0.1:39962/
[junit4] 2> 239303 WARN (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.SocketProxy Closing 0 connections to: http://127.0.0.1:45871/, target: http://127.0.0.1:34721/
[junit4] 2> 239303 WARN (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.SocketProxy Closing 16 connections to: http://127.0.0.1:38214/, target: http://127.0.0.1:35898/
[junit4] 2> 239303 WARN (TEST-LeaderFailoverAfterPartitionTest.test-seed#[7634F286EDC1E95]) [ ] o.a.s.c.SocketProxy Closing 6 connections to: http://127.0.0.1:37992/, target: http://127.0.0.1:43308/
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=LeaderFailoverAfterPartitionTest -Dtests.method=test -Dtests.seed=7634F286EDC1E95 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-SG -Dtests.timezone=Pacific/Honolulu -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 151s J0 | LeaderFailoverAfterPartitionTest.test <<<
[junit4] > Throwable #1: java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node2:{"core":"c8n_1x3_lf_shard1_replica3","base_url":"http://127.0.0.1:38214","node_name":"127.0.0.1:38214_","state":"active","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/16)={
[junit4] > "replicationFactor":"3",
[junit4] > "shards":{"shard1":{
[junit4] > "range":"80000000-7fffffff",
[junit4] > "state":"active",
[junit4] > "replicas":{
[junit4] > "core_node1":{
[junit4] > "state":"down",
[junit4] > "base_url":"http://127.0.0.1:38070",
[junit4] > "core":"c8n_1x3_lf_shard1_replica1",
[junit4] > "node_name":"127.0.0.1:38070_"},
[junit4] > "core_node2":{
[junit4] > "core":"c8n_1x3_lf_shard1_replica3",
[junit4] > "base_url":"http://127.0.0.1:38214",
[junit4] > "node_name":"127.0.0.1:38214_",
[junit4] > "state":"active",
[junit4] > "leader":"true"},
[junit4] > "core_node3":{
[junit4] > "core":"c8n_1x3_lf_shard1_replica2",
[junit4] > "base_url":"http://127.0.0.1:45871",
[junit4] > "node_name":"127.0.0.1:45871_",
[junit4] > "state":"down"}}}},
[junit4] > "router":{"name":"compositeId"},
[junit4] > "maxShardsPerNode":"1",
[junit4] > "autoAddReplicas":"false"}
[junit4] > at __randomizedtesting.SeedInfo.seed([7634F286EDC1E95:8F3770F2C020736D]:0)
[junit4] > at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:168)
[junit4] > at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:55)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 239306 INFO (SUITE-LeaderFailoverAfterPartitionTest-seed#[7634F286EDC1E95]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.LeaderFailoverAfterPartitionTest_7634F286EDC1E95-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62): {range_facet_l_dv=PostingsFormat(name=Asserting), multiDefault=FST50, a_t=FST50, intDefault=PostingsFormat(name=MockRandom), id=PostingsFormat(name=Asserting), range_facet_i_dv=FST50, text=PostingsFormat(name=Memory doPackFST= true), range_facet_l=FST50, timestamp=FST50}, docValues:{range_facet_l_dv=DocValuesFormat(name=Memory), _version_=DocValuesFormat(name=Lucene54), range_facet_i_dv=DocValuesFormat(name=Direct), intDvoDefault=DocValuesFormat(name=Memory), timestamp=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=306, maxMBSortInHeap=5.592635716576553, sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=en-SG, timezone=Pacific/Honolulu
[junit4] 2> NOTE: Linux 4.4.0-53-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=12,threads=1,free=139562456,total=526385152
[junit4] 2> NOTE: All tests run in this JVM: [LeaderInitiatedRecoveryOnShardRestartTest, BlockCacheTest, TestClusterProperties, CollectionsAPISolrJTest, OverseerCollectionConfigSetProcessorTest, SolrCoreTest, DistributedSpellCheckComponentTest, BitVectorTest, LeaderFailoverAfterPartitionTest]
[junit4] Completed [95/696 (1!)] on J0 in 150.63s, 1 test, 1 failure <<< FAILURES!
[...truncated 64593 lines...]
[JENKINS] Lucene-Solr-6.x-Linux (64bit/jdk1.8.0_121) - Build # 2893
- Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/2893/
Java: 64bit/jdk1.8.0_121 -XX:-UseCompressedOops -XX:+UseSerialGC
1 tests failed.
FAILED: org.apache.solr.cloud.LeaderFailureAfterFreshStartTest.test
Error Message:
timeout waiting to see all nodes active
Stack Trace:
java.lang.AssertionError: timeout waiting to see all nodes active
at __randomizedtesting.SeedInfo.seed([5BB34B1F6DD7462:8DEF0B6B5821199A]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.LeaderFailureAfterFreshStartTest.waitTillNodesActive(LeaderFailureAfterFreshStartTest.java:233)
at org.apache.solr.cloud.LeaderFailureAfterFreshStartTest.restartNodes(LeaderFailureAfterFreshStartTest.java:173)
at org.apache.solr.cloud.LeaderFailureAfterFreshStartTest.test(LeaderFailureAfterFreshStartTest.java:148)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 12629 lines...]
[junit4] Suite: org.apache.solr.cloud.LeaderFailureAfterFreshStartTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/init-core-data-001
[junit4] 2> 1444313 INFO (SUITE-LeaderFailureAfterFreshStartTest-seed#[5BB34B1F6DD7462]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields
[junit4] 2> 1444314 INFO (SUITE-LeaderFailureAfterFreshStartTest-seed#[5BB34B1F6DD7462]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 1444315 INFO (SUITE-LeaderFailureAfterFreshStartTest-seed#[5BB34B1F6DD7462]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /jv_f/v
[junit4] 2> 1444317 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1444317 INFO (Thread-2450) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1444317 INFO (Thread-2450) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1444417 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.ZkTestServer start zk server on port:45850
[junit4] 2> 1444422 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1444423 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 1444424 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1444425 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1444425 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1444426 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1444426 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1444427 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1444427 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1444428 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1444428 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1444491 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/control-001/cores/collection1
[junit4] 2> 1444492 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 1444493 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6d52335c{/jv_f/v,null,AVAILABLE}
[junit4] 2> 1444494 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@6bce4929{SSL,[ssl, http/1.1]}{127.0.0.1:43506}
[junit4] 2> 1444494 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.Server Started @1446624ms
[junit4] 2> 1444494 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/tempDir-001/control/data, hostContext=/jv_f/v, hostPort=43506, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/control-001/cores}
[junit4] 2> 1444494 ERROR (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1444495 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 6.5.0
[junit4] 2> 1444495 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1444495 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1444495 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-19T14:59:10.395Z
[junit4] 2> 1444497 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1444497 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/control-001/solr.xml
[junit4] 2> 1444525 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 1444526 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45850/solr
[junit4] 2> 1444535 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1444536 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:43506_jv_f%2Fv
[junit4] 2> 1444536 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.Overseer Overseer (id=97485871527886853-127.0.0.1:43506_jv_f%2Fv-n_0000000000) starting
[junit4] 2> 1444539 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43506_jv_f%2Fv
[junit4] 2> 1444540 INFO (zkCallback-1835-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1444629 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/control-001/cores
[junit4] 2> 1444629 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 1444631 INFO (OverseerStateUpdate-97485871527886853-127.0.0.1:43506_jv_f%2Fv-n_0000000000) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1445638 WARN (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 1445639 INFO (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
[junit4] 2> 1445650 INFO (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 1445729 WARN (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 1445731 INFO (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 1445737 INFO (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 1445737 INFO (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/control-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/control-001/cores/collection1/data/]
[junit4] 2> 1445737 INFO (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5197f4c9
[junit4] 2> 1445739 INFO (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=63.8974609375, floorSegmentMB=1.9150390625, forceMergeDeletesPctAllowed=26.36014600709151, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7800203775232898
[junit4] 2> 1445752 WARN (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1445778 INFO (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1445778 INFO (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1445778 INFO (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1445778 INFO (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1445779 INFO (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=25, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7524767408330313]
[junit4] 2> 1445779 INFO (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@74a20e56[collection1] main]
[junit4] 2> 1445780 INFO (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1445780 INFO (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1445780 INFO (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1445781 INFO (searcherExecutor-5699-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv x:collection1 c:control_collection) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@74a20e56[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1445782 INFO (coreLoadExecutor-5698-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559773945981304832
[junit4] 2> 1445785 INFO (coreZkRegister-5691-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv x:collection1 c:control_collection) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1445785 INFO (coreZkRegister-5691-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv x:collection1 c:control_collection) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1445785 INFO (coreZkRegister-5691-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv x:collection1 c:control_collection) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:43506/jv_f/v/collection1/
[junit4] 2> 1445785 INFO (coreZkRegister-5691-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv x:collection1 c:control_collection) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1445785 INFO (coreZkRegister-5691-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv x:collection1 c:control_collection) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:43506/jv_f/v/collection1/ has no replicas
[junit4] 2> 1445787 INFO (coreZkRegister-5691-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv x:collection1 c:control_collection) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:43506/jv_f/v/collection1/ shard1
[junit4] 2> 1445938 INFO (coreZkRegister-5691-thread-1-processing-n:127.0.0.1:43506_jv_f%2Fv x:collection1 c:control_collection) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1446012 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1446012 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45850/solr ready
[junit4] 2> 1446012 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1446012 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
[junit4] 2> 1446075 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-1-001/cores/collection1
[junit4] 2> 1446075 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-1-001
[junit4] 2> 1446076 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 1446077 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7c41c2f4{/jv_f/v,null,AVAILABLE}
[junit4] 2> 1446078 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@6c0037a4{SSL,[ssl, http/1.1]}{127.0.0.1:38154}
[junit4] 2> 1446078 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.Server Started @1448208ms
[junit4] 2> 1446078 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/jv_f/v, hostPort=38154, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-1-001/cores}
[junit4] 2> 1446078 ERROR (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1446079 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 6.5.0
[junit4] 2> 1446079 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1446079 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1446079 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-19T14:59:11.979Z
[junit4] 2> 1446081 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1446081 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-1-001/solr.xml
[junit4] 2> 1446085 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 1446086 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45850/solr
[junit4] 2> 1446089 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1446090 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1446091 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38154_jv_f%2Fv
[junit4] 2> 1446093 INFO (zkCallback-1845-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1446093 INFO (zkCallback-1839-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1446093 INFO (zkCallback-1835-thread-3-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1446146 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-1-001/cores
[junit4] 2> 1446146 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 1446148 INFO (OverseerStateUpdate-97485871527886853-127.0.0.1:43506_jv_f%2Fv-n_0000000000) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1446249 WARN (OverseerStateUpdate-97485871527886853-127.0.0.1:43506_jv_f%2Fv-n_0000000000) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.Overseer Bad version writing to ZK using compare-and-set, will force refresh cluster state: KeeperErrorCode = BadVersion for /collections/collection1/state.json
[junit4] 2> 1446250 INFO (zkCallback-1845-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1446251 INFO (OverseerStateUpdate-97485871527886853-127.0.0.1:43506_jv_f%2Fv-n_0000000000) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1446251 INFO (zkCallback-1845-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1447155 WARN (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 1447155 INFO (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
[junit4] 2> 1447166 INFO (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 1447248 WARN (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 1447250 INFO (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 1447256 INFO (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1447256 INFO (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-1-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-1-001/cores/collection1/data/]
[junit4] 2> 1447256 INFO (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5197f4c9
[junit4] 2> 1447257 INFO (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=63.8974609375, floorSegmentMB=1.9150390625, forceMergeDeletesPctAllowed=26.36014600709151, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7800203775232898
[junit4] 2> 1447304 WARN (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1447329 INFO (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1447329 INFO (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1447330 INFO (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1447330 INFO (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1447330 INFO (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=25, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7524767408330313]
[junit4] 2> 1447331 INFO (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@135af876[collection1] main]
[junit4] 2> 1447331 INFO (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1447331 INFO (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1447331 INFO (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1447332 INFO (searcherExecutor-5710-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv x:collection1 c:collection1) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@135af876[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1447333 INFO (coreLoadExecutor-5709-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559773947607646208
[junit4] 2> 1447336 INFO (coreZkRegister-5704-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv x:collection1 c:collection1) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1447336 INFO (coreZkRegister-5704-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv x:collection1 c:collection1) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1447336 INFO (coreZkRegister-5704-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv x:collection1 c:collection1) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:38154/jv_f/v/collection1/
[junit4] 2> 1447336 INFO (coreZkRegister-5704-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv x:collection1 c:collection1) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1447336 INFO (coreZkRegister-5704-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv x:collection1 c:collection1) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:38154/jv_f/v/collection1/ has no replicas
[junit4] 2> 1447337 INFO (coreZkRegister-5704-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv x:collection1 c:collection1) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:38154/jv_f/v/collection1/ shard1
[junit4] 2> 1447439 INFO (zkCallback-1845-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1447488 INFO (coreZkRegister-5704-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv x:collection1 c:collection1) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1447562 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-2-001/cores/collection1
[junit4] 2> 1447563 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-2-001
[junit4] 2> 1447564 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 1447565 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7ecb5ac{/jv_f/v,null,AVAILABLE}
[junit4] 2> 1447565 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@69dc7533{SSL,[ssl, http/1.1]}{127.0.0.1:34747}
[junit4] 2> 1447565 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.Server Started @1449696ms
[junit4] 2> 1447565 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/jv_f/v, hostPort=34747, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-2-001/cores}
[junit4] 2> 1447566 ERROR (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1447566 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 6.5.0
[junit4] 2> 1447566 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1447566 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1447566 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-19T14:59:13.466Z
[junit4] 2> 1447568 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1447568 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-2-001/solr.xml
[junit4] 2> 1447572 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 1447573 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45850/solr
[junit4] 2> 1447577 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:34747_jv_f%2Fv ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 1447577 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:34747_jv_f%2Fv ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1447578 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:34747_jv_f%2Fv ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34747_jv_f%2Fv
[junit4] 2> 1447578 INFO (zkCallback-1835-thread-3-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1447579 INFO (zkCallback-1851-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1447579 INFO (zkCallback-1845-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1447579 INFO (zkCallback-1839-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1447579 INFO (zkCallback-1845-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1447646 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:34747_jv_f%2Fv ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-2-001/cores
[junit4] 2> 1447646 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:34747_jv_f%2Fv ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 1447648 INFO (OverseerStateUpdate-97485871527886853-127.0.0.1:43506_jv_f%2Fv-n_0000000000) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1447749 WARN (OverseerStateUpdate-97485871527886853-127.0.0.1:43506_jv_f%2Fv-n_0000000000) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.Overseer Bad version writing to ZK using compare-and-set, will force refresh cluster state: KeeperErrorCode = BadVersion for /collections/collection1/state.json
[junit4] 2> 1447750 INFO (zkCallback-1851-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1447750 INFO (zkCallback-1845-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1447751 INFO (OverseerStateUpdate-97485871527886853-127.0.0.1:43506_jv_f%2Fv-n_0000000000) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1447751 INFO (zkCallback-1851-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1447751 INFO (zkCallback-1845-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1448655 WARN (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 1448656 INFO (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
[junit4] 2> 1448667 INFO (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 1448750 WARN (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 1448751 INFO (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 1448757 INFO (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1448757 INFO (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-2-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-2-001/cores/collection1/data/]
[junit4] 2> 1448757 INFO (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5197f4c9
[junit4] 2> 1448758 INFO (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=63.8974609375, floorSegmentMB=1.9150390625, forceMergeDeletesPctAllowed=26.36014600709151, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7800203775232898
[junit4] 2> 1448795 WARN (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1448822 INFO (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1448823 INFO (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1448823 INFO (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1448823 INFO (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1448824 INFO (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=25, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7524767408330313]
[junit4] 2> 1448824 INFO (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@50b04a49[collection1] main]
[junit4] 2> 1448824 INFO (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1448825 INFO (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1448825 INFO (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1448826 INFO (searcherExecutor-5721-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv x:collection1 c:collection1) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@50b04a49[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1448826 INFO (coreLoadExecutor-5720-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559773949173170176
[junit4] 2> 1448829 INFO (coreZkRegister-5715-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv x:collection1 c:collection1) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
[junit4] 2> 1448829 INFO (updateExecutor-1848-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 1448829 INFO (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 1448829 INFO (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 1448830 INFO (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
[junit4] 2> 1448830 INFO (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 1448830 INFO (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [https://127.0.0.1:38154/jv_f/v/collection1/] and I am [https://127.0.0.1:34747/jv_f/v/collection1/]
[junit4] 2> 1448831 INFO (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [https://127.0.0.1:38154/jv_f/v]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:34747_jv_f%252Fv&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 1448886 INFO (qtp153843608-11015) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 1448886 INFO (qtp153843608-11015) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
[junit4] 2> 1448886 INFO (qtp153843608-11015) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:34747_jv_f%2Fv, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"https://127.0.0.1:34747/jv_f/v","node_name":"127.0.0.1:34747_jv_f%2Fv","state":"down"}
[junit4] 2> 1448932 INFO (zkCallback-1845-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1448932 INFO (zkCallback-1851-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1449078 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-3-001/cores/collection1
[junit4] 2> 1449078 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-3-001
[junit4] 2> 1449080 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 1449080 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@23f7cc6e{/jv_f/v,null,AVAILABLE}
[junit4] 2> 1449081 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@643d2473{SSL,[ssl, http/1.1]}{127.0.0.1:39272}
[junit4] 2> 1449081 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.Server Started @1451211ms
[junit4] 2> 1449081 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/jv_f/v, hostPort=39272, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-3-001/cores}
[junit4] 2> 1449082 ERROR (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1449082 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 6.5.0
[junit4] 2> 1449082 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1449082 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1449082 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-19T14:59:14.982Z
[junit4] 2> 1449084 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1449084 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-3-001/solr.xml
[junit4] 2> 1449088 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 1449088 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45850/solr
[junit4] 2> 1449094 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:39272_jv_f%2Fv ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 1449094 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:39272_jv_f%2Fv ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1449095 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:39272_jv_f%2Fv ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39272_jv_f%2Fv
[junit4] 2> 1449096 INFO (zkCallback-1851-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1449096 INFO (zkCallback-1845-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1449096 INFO (zkCallback-1839-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1449096 INFO (zkCallback-1858-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1449096 INFO (zkCallback-1835-thread-3-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1449133 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:39272_jv_f%2Fv ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-3-001/cores
[junit4] 2> 1449133 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [n:127.0.0.1:39272_jv_f%2Fv ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 1449134 INFO (OverseerStateUpdate-97485871527886853-127.0.0.1:43506_jv_f%2Fv-n_0000000000) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1449236 INFO (zkCallback-1851-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1449236 INFO (zkCallback-1858-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1449236 INFO (zkCallback-1845-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1449886 INFO (qtp153843608-11015) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:34747_jv_f%2Fv, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"https://127.0.0.1:34747/jv_f/v","node_name":"127.0.0.1:34747_jv_f%2Fv","state":"recovering"}
[junit4] 2> 1449887 INFO (qtp153843608-11015) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 1449887 INFO (qtp153843608-11015) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:34747_jv_f%252Fv&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1000
[junit4] 2> 1450142 WARN (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 1450143 INFO (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
[junit4] 2> 1450160 INFO (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 1450279 WARN (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 1450282 INFO (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 1450302 INFO (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1450302 INFO (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-3-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001/shard-3-001/cores/collection1/data/]
[junit4] 2> 1450302 INFO (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5197f4c9
[junit4] 2> 1450305 INFO (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=63.8974609375, floorSegmentMB=1.9150390625, forceMergeDeletesPctAllowed=26.36014600709151, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7800203775232898
[junit4] 2> 1450330 WARN (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1450357 INFO (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1450357 INFO (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1450358 INFO (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1450358 INFO (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1450358 INFO (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=25, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7524767408330313]
[junit4] 2> 1450358 INFO (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@396b78db[collection1] main]
[junit4] 2> 1450359 INFO (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1450359 INFO (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1450359 INFO (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1450360 INFO (searcherExecutor-5732-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv x:collection1 c:collection1) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@396b78db[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1450360 INFO (coreLoadExecutor-5731-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559773950781685760
[junit4] 2> 1450363 INFO (coreZkRegister-5726-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv x:collection1 c:collection1) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
[junit4] 2> 1450364 INFO (updateExecutor-1855-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 1450364 INFO (recoveryExecutor-1856-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 1450364 INFO (recoveryExecutor-1856-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 1450364 INFO (recoveryExecutor-1856-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
[junit4] 2> 1450364 INFO (recoveryExecutor-1856-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 1450364 INFO (recoveryExecutor-1856-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [https://127.0.0.1:38154/jv_f/v/collection1/] and I am [https://127.0.0.1:39272/jv_f/v/collection1/]
[junit4] 2> 1450366 INFO (recoveryExecutor-1856-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [https://127.0.0.1:38154/jv_f/v]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:39272_jv_f%252Fv&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 1450368 INFO (qtp153843608-11016) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 1450368 INFO (qtp153843608-11016) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
[junit4] 2> 1450368 INFO (qtp153843608-11016) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:39272_jv_f%2Fv, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"https://127.0.0.1:39272/jv_f/v","node_name":"127.0.0.1:39272_jv_f%2Fv","state":"down"}
[junit4] 2> 1450466 INFO (zkCallback-1845-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1450466 INFO (zkCallback-1858-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1450466 INFO (zkCallback-1851-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1450508 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.SolrTestCaseJ4 ###Starting test
[junit4] 2> 1450509 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.LeaderFailureAfterFreshStartTest Leader node_name: core_node1, url: https://127.0.0.1:38154/jv_f/v/collection1
[junit4] 2> 1450509 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.LeaderFailureAfterFreshStartTest Nonleader node_name: core_node2, url: https://127.0.0.1:34747/jv_f/v/collection1
[junit4] 2> 1450509 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.LeaderFailureAfterFreshStartTest Nonleader node_name: core_node3, url: https://127.0.0.1:39272/jv_f/v/collection1
[junit4] 2> 1450509 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.ChaosMonkey monkey: kill shard! 39272
[junit4] 2> 1450509 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@643d2473{SSL,[ssl, http/1.1]}{127.0.0.1:0}
[junit4] 2> 1450509 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1783636152
[junit4] 2> 1450510 WARN (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection1] coreNodeName=[core_node3]
[junit4] 2> 1450511 ERROR (recoveryExecutor-1856-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Error while trying to recover. core=collection1:java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:38154/jv_f/v
[junit4] 2> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
[junit4] 2> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
[junit4] 2> at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:614)
[junit4] 2> at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:591)
[junit4] 2> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:351)
[junit4] 2> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222)
[junit4] 2> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:38154/jv_f/v
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:624)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.lambda$httpUriRequest$0(HttpSolrClient.java:319)
[junit4] 2> ... 5 more
[junit4] 2> Caused by: java.net.SocketException: Socket closed
[junit4] 2> at java.net.SocketInputStream.read(SocketInputStream.java:204)
[junit4] 2> at java.net.SocketInputStream.read(SocketInputStream.java:141)
[junit4] 2> at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
[junit4] 2> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
[junit4] 2> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
[junit4] 2> at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
[junit4] 2> at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
[junit4] 2> at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
[junit4] 2> at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
[junit4] 2> at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
[junit4] 2> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
[junit4] 2> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
[junit4] 2> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
[junit4] 2> at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
[junit4] 2> at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
[junit4] 2> at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
[junit4] 2> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
[junit4] 2> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
[junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
[junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
[junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
[junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
[junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
[junit4] 2> ... 6 more
[junit4] 2>
[junit4] 2> 1450511 INFO (recoveryExecutor-1856-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Replay not started, or was not successful... still buffering updates.
[junit4] 2> 1450511 INFO (recoveryExecutor-1856-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy RecoveryStrategy has been closed
[junit4] 2> 1450511 INFO (recoveryExecutor-1856-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Finished recovery process, successful=[false]
[junit4] 2> 1450512 INFO (coreCloseExecutor-5737-thread-1) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.SolrCore [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@77da682f
[junit4] 2> 1450512 WARN (coreCloseExecutor-5737-thread-1) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection1] coreNodeName=[core_node3]
[junit4] 2> 1450547 INFO (coreCloseExecutor-5737-thread-1) [n:127.0.0.1:39272_jv_f%2Fv c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
[junit4] 2> 1450548 INFO (zkCallback-1851-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 1450548 INFO (zkCallback-1845-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 1450548 INFO (zkCallback-1835-thread-3-processing-n:127.0.0.1:43506_jv_f%2Fv) [n:127.0.0.1:43506_jv_f%2Fv ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 1450548 INFO (zkCallback-1839-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 1450615 INFO (zkCallback-1851-thread-1-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1450615 INFO (zkCallback-1845-thread-1-processing-n:127.0.0.1:38154_jv_f%2Fv) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1451369 INFO (qtp153843608-11016) [n:127.0.0.1:38154_jv_f%2Fv ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=false, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:39272_jv_f%2Fv, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"https://127.0.0.1:39272/jv_f/v","node_name":"127.0.0.1:39272_jv_f%2Fv","state":"down"}
[junit4] 2> 1452048 WARN (zkCallback-1858-thread-1-processing-n:127.0.0.1:39272_jv_f%2Fv) [n:127.0.0.1:39272_jv_f%2Fv ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 1452048 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 1452048 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@23f7cc6e{/jv_f/v,null,UNAVAILABLE}
[junit4] 2> 1452049 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.ChaosMonkey monkey: stop shard! 39272
[junit4] 2> 1452102 INFO (qtp1840585938-10976) [n:127.0.0.1:43506_jv_f%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/jv_f/v path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1559773952607256576)} 0 1
[junit4] 2> 1452214 INFO (qtp1960868147-11043) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/jv_f/v path=/update params={update.distrib=FROMLEADER&_version_=-1559773952662831104&distrib.from=https://127.0.0.1:38154/jv_f/v/collection1/&wt=javabin&version=2}{deleteByQuery=*:* (-1559773952662831104)} 0 0
[junit4] 2> 1452215 INFO (qtp153843608-11020) [n:127.0.0.1:38154_jv_f%2Fv c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/jv_f/v path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1559773952662831104)} 0 61
[junit4] 2> 1452215 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Wait for recoveries to finish - wait 30 for each attempt
[junit4] 2> 1452215 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
[junit4] 2> 1452369
[...truncated too long message...]
ack-1865-thread-4-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SolrCore [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@4b4ff40d
[junit4] 2> 1699102 INFO (zkCallback-1865-thread-4-processing-n:127.0.0.1:34747_jv_f%2Fv) [n:127.0.0.1:34747_jv_f%2Fv c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
[junit4] 2> 1699102 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 1699103 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@845b2af{SSL,[ssl, http/1.1]}{127.0.0.1:34747}
[junit4] 2> 1699103 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@454fab3b{/jv_f/v,null,UNAVAILABLE}
[junit4] 2> 1699103 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.ChaosMonkey monkey: stop shard! 39272
[junit4] 2> 1699104 INFO (TEST-LeaderFailureAfterFreshStartTest.test-seed#[5BB34B1F6DD7462]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:45850 45850
[junit4] 2> 1699115 INFO (Thread-2450) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:45850 45850
[junit4] 2> 1699115 WARN (Thread-2450) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/aliases.json
[junit4] 2> 5 /solr/security.json
[junit4] 2> 5 /solr/configs/conf1
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/clusterstate.json
[junit4] 2> 6 /solr/clusterprops.json
[junit4] 2> 4 /solr/collections/collection1/state.json
[junit4] 2> 2 /solr/overseer_elect/election/97485871527886858-127.0.0.1:38154_jv_f%2Fv-n_0000000001
[junit4] 2> 2 /solr/collections/collection1/leader_elect/shard1/election/97485871527886858-core_node1-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 252 /solr/overseer/collection-queue-work
[junit4] 2> 18 /solr/overseer/queue
[junit4] 2> 13 /solr/overseer/queue-work
[junit4] 2> 6 /solr/collections
[junit4] 2> 5 /solr/live_nodes
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=LeaderFailureAfterFreshStartTest -Dtests.method=test -Dtests.seed=5BB34B1F6DD7462 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=el-GR -Dtests.timezone=Pacific/Noumea -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 255s J1 | LeaderFailureAfterFreshStartTest.test <<<
[junit4] > Throwable #1: java.lang.AssertionError: timeout waiting to see all nodes active
[junit4] > at __randomizedtesting.SeedInfo.seed([5BB34B1F6DD7462:8DEF0B6B5821199A]:0)
[junit4] > at org.apache.solr.cloud.LeaderFailureAfterFreshStartTest.waitTillNodesActive(LeaderFailureAfterFreshStartTest.java:233)
[junit4] > at org.apache.solr.cloud.LeaderFailureAfterFreshStartTest.restartNodes(LeaderFailureAfterFreshStartTest.java:173)
[junit4] > at org.apache.solr.cloud.LeaderFailureAfterFreshStartTest.test(LeaderFailureAfterFreshStartTest.java:148)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1699118 INFO (SUITE-LeaderFailureAfterFreshStartTest-seed#[5BB34B1F6DD7462]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailureAfterFreshStartTest_5BB34B1F6DD7462-001
[junit4] 2> Feb 19, 2017 3:03:25 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62): {other_tl1=PostingsFormat(name=Memory doPackFST= false), rnd_s=PostingsFormat(name=Memory doPackFST= false), multiDefault=FSTOrd50, a_t=FSTOrd50, a_i1=FSTOrd50, id=Lucene50(blocksize=128), text=PostingsFormat(name=Direct), timestamp=FSTOrd50}, docValues:{range_facet_l_dv=DocValuesFormat(name=Memory), _version_=DocValuesFormat(name=Lucene54), intDefault=DocValuesFormat(name=Lucene54), range_facet_i_dv=DocValuesFormat(name=Direct), intDvoDefault=DocValuesFormat(name=Memory), range_facet_l=DocValuesFormat(name=Direct), timestamp=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=1595, maxMBSortInHeap=7.550076415712336, sim=RandomSimilarity(queryNorm=false,coord=yes): {}, locale=el-GR, timezone=Pacific/Noumea
[junit4] 2> NOTE: Linux 4.4.0-53-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=12,threads=1,free=199731992,total=514445312
[junit4] 2> NOTE: All tests run in this JVM: [TestDynamicFieldCollectionResource, URLClassifyProcessorTest, TestPartialUpdateDeduplication, MetricsHandlerTest, IndexSchemaTest, CollectionTooManyReplicasTest, TestCloudPseudoReturnFields, TestWordDelimiterFilterFactory, TestUninvertingReader, DirectSolrSpellCheckerTest, TestFilteredDocIdSet, MigrateRouteKeyTest, ActionThrottleTest, SimplePostToolTest, TestPointFields, TestQueryUtils, TestSlowCompositeReaderWrapper, SpellCheckCollatorTest, TestManagedSchema, ExplicitHLLTest, TestInfoStreamLogging, TestLegacyFieldCache, TestConfigsApi, DistributedIntervalFacetingTest, TestXmlQParserPlugin, CollectionsAPIDistributedZkTest, TestRequestStatusCollectionAPI, DistributedFacetPivotWhiteBoxTest, TermVectorComponentTest, BlockDirectoryTest, SolrGangliaReporterTest, CSVRequestHandlerTest, ChangedSchemaMergeTest, TestEmbeddedSolrServerConstructors, TestSurroundQueryParser, UpdateRequestProcessorFactoryTest, SchemaVersionSpecificBehaviorTest, TestNoOpRegenerator, TestFuzzyAnalyzedSuggestions, DocValuesTest, DistributedMLTComponentTest, TestSubQueryTransformerCrossCore, TestComplexPhraseQParserPlugin, StatsReloadRaceTest, TestMacros, DocumentBuilderTest, TestDistributedGrouping, TestSchemaVersionResource, SuggesterFSTTest, AsyncCallRequestStatusResponseTest, TestLRUCache, TestReloadAndDeleteDocs, TestRandomCollapseQParserPlugin, ExitableDirectoryReaderTest, SolrMetricManagerTest, HttpPartitionTest, TestAnalyzeInfixSuggestions, EnumFieldTest, FileUtilsTest, TestSweetSpotSimilarityFactory, TestStressVersions, TestSchemaNameResource, CdcrUpdateLogTest, TestFastOutputStream, TestSolrQueryParserResource, RestartWhileUpdatingTest, TestFieldCacheReopen, SliceStateTest, ZkCLITest, ShardRoutingTest, BasicAuthIntegrationTest, JsonLoaderTest, TestRTGBase, TestManagedResourceStorage, TestExpandComponent, RankQueryTest, DateMathParserTest, HighlighterTest, TestSQLHandlerNonCloud, TestLeaderInitiatedRecoveryThread, TestCharFilters, HLLUtilTest, HdfsChaosMonkeyNothingIsSafeTest, DirectSolrConnectionTest, IndexBasedSpellCheckerTest, DeleteLastCustomShardedReplicaTest, TestConfigReload, TestSimpleQParserPlugin, SharedFSAutoReplicaFailoverUtilsTest, JsonValidatorTest, DistributedTermsComponentTest, TestPushWriter, MultiTermTest, RemoteQueryErrorTest, DistanceFunctionTest, HdfsTlogReplayBufferedWhileIndexingTest, TestInitQParser, SortByFunctionTest, AnalyticsQueryTest, ClusterStateUpdateTest, TestStressInPlaceUpdates, DistributedSuggestComponentTest, TestLMDirichletSimilarityFactory, DistributedVersionInfoTest, TestGeoJSONResponseWriter, OverseerTaskQueueTest, SaslZkACLProviderTest, BasicFunctionalityTest, BigEndianAscendingWordDeserializerTest, TestBinaryResponseWriter, TolerantUpdateProcessorTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, SyncSliceTest, TestRandomFaceting, LeaderElectionTest, TestZkChroot, ShardRoutingCustomTest, TestRecovery, TestHashPartitioner, TestStressReorder, TestReload, HardAutoCommitTest, StatsComponentTest, SolrCmdDistributorTest, TestFiltering, CurrencyFieldXmlFileTest, SolrIndexSplitterTest, TestCoreDiscovery, TestExtendedDismaxParser, SolrRequestParserTest, TestTrie, SpatialFilterTest, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, QueryEqualityTest, StatelessScriptUpdateProcessorFactoryTest, TestSolrDeletionPolicy1, CacheHeaderTest, StandardRequestHandlerTest, TestWriterPerf, TestQueryTypes, TestOmitPositions, TermsComponentTest, TestValueSourceCache, MoreLikeThisHandlerTest, TestSolrQueryParser, LoggingHandlerTest, JSONWriterTest, QueryParsingTest, BinaryUpdateRequestHandlerTest, TestQuerySenderListener, AlternateDirectoryTest, TestSolrIndexConfig, CopyFieldTest, BadComponentTest, TestMergePolicyConfig, TestSearchPerf, NumericFieldsTest, MinimalSchemaTest, TestSolrCoreProperties, TestLuceneMatchVersion, EchoParamsTest, TestDFRSimilarityFactory, TestBM25SimilarityFactory, ResourceLoaderTest, TestFastLRUCache, PreAnalyzedFieldTest, TestSystemIdResolver, TestSuggestSpellingConverter, SpellingQueryConverterTest, ClusterStateTest, ZkNodePropsTest, CursorPagingTest, TestPathTrie, AliasIntegrationTest, CdcrReplicationDistributedZkTest, CdcrReplicationHandlerTest, CdcrVersionReplicationTest, CloudExitableDirectoryReaderTest, CollectionReloadTest, CollectionStateFormat2Test, DeleteNodeTest, DeleteShardTest, DistribJoinFromCollectionTest, DistributedQueueTest, DocValuesNotIndexedTest, LeaderElectionContextKeyTest, LeaderFailureAfterFreshStartTest]
[junit4] Completed [531/696 (1!)] on J1 in 255.82s, 1 test, 1 failure <<< FAILURES!
[...truncated 63230 lines...]
[JENKINS] Lucene-Solr-6.x-Linux (32bit/jdk1.8.0_121) - Build # 2892
- Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/2892/
Java: 32bit/jdk1.8.0_121 -client -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.update.TestInPlaceUpdatesDistrib.test
Error Message:
Thread pool didn't terminate within 10 secs
Stack Trace:
java.lang.AssertionError: Thread pool didn't terminate within 10 secs
at __randomizedtesting.SeedInfo.seed([9A6C0179F16FF134:12383EA35F939CCC]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.update.TestInPlaceUpdatesDistrib.reorderedDBQsWithInPlaceUpdatesShouldNotThrowReplicaInLIRTest(TestInPlaceUpdatesDistrib.java:740)
at org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:145)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11868 lines...]
[junit4] Suite: org.apache.solr.update.TestInPlaceUpdatesDistrib
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/init-core-data-001
[junit4] 2> 680990 INFO (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields
[junit4] 2> 680992 INFO (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 680992 INFO (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /ok/
[junit4] 2> 680993 INFO (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore
[junit4] 2> 680994 INFO (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/lib, /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/lib/classes]
[junit4] 2> 681018 INFO (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
[junit4] 2> 681030 INFO (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.s.IndexSchema [null] Schema name=inplace-updates
[junit4] 2> 681033 INFO (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
[junit4] 2> 681036 INFO (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
[junit4] 2> 681057 WARN (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
[junit4] 2> 681060 INFO (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa2421
[junit4] 2> 681060 WARN (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
[junit4] 2> 681065 INFO (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa2421
[junit4] 2> 681065 WARN (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
[junit4] 2> 681065 INFO (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa2421
[junit4] 2> 681066 INFO (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/lib, /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/lib/classes]
[junit4] 2> 681094 INFO (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
[junit4] 2> 681107 INFO (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
[junit4] 2> 681111 INFO (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
[junit4] 2> 681113 INFO (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1
[junit4] 2> 681113 WARN (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
[junit4] 2> 681114 INFO (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa2421
[junit4] 2> 681114 INFO (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/init-core-data-001/]
[junit4] 2> 681115 INFO (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa2421
[junit4] 2> 681124 WARN (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 681180 INFO (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 681180 INFO (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 681182 INFO (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 681182 INFO (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 681183 INFO (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@89f447[collection1] main]
[junit4] 2> 681184 WARN (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.r.ManagedResourceStorage Cannot write to config directory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf; switching to use InMemory storage instead.
[junit4] 2> 681184 INFO (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 681187 INFO (searcherExecutor-3162-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@89f447[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 681187 INFO (coreLoadExecutor-3161-thread-1) [ x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559763923443384320
[junit4] 2> 681190 INFO (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore end
[junit4] 2> 681194 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 681194 INFO (Thread-1387) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 681194 INFO (Thread-1387) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 681294 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.ZkTestServer start zk server on port:37981
[junit4] 2> 681315 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 681317 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema-inplace-updates.xml to /configs/conf1/schema.xml
[junit4] 2> 681319 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 681320 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 681321 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 681332 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 681334 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 681335 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 681336 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 681337 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 681339 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 681430 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/control-001/cores/collection1
[junit4] 2> 681434 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 681435 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1bdd22d{/ok,null,AVAILABLE}
[junit4] 2> 681436 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@15795bc{SSL,[ssl, http/1.1]}{127.0.0.1:40734}
[junit4] 2> 681436 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.Server Started @682759ms
[junit4] 2> 681436 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/tempDir-001/control/data, hostContext=/ok, hostPort=40734, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/control-001/cores}
[junit4] 2> 681437 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 681437 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.5.0
[junit4] 2> 681437 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 681437 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 681437 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-19T12:19:53.695Z
[junit4] 2> 681440 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 681440 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/control-001/solr.xml
[junit4] 2> 681447 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 681448 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37981/solr
[junit4] 2> 681463 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:40734_ok ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 681464 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:40734_ok ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:40734_ok
[junit4] 2> 681465 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:40734_ok ] o.a.s.c.Overseer Overseer (id=97485245215735813-127.0.0.1:40734_ok-n_0000000000) starting
[junit4] 2> 681472 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:40734_ok ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40734_ok
[junit4] 2> 681473 INFO (zkCallback-3641-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 681940 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:40734_ok ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/control-001/cores
[junit4] 2> 681940 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:40734_ok ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 681943 INFO (OverseerStateUpdate-97485245215735813-127.0.0.1:40734_ok-n_0000000000) [n:127.0.0.1:40734_ok ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 682956 INFO (coreLoadExecutor-3174-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok c:control_collection x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
[junit4] 2> 682968 INFO (coreLoadExecutor-3174-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
[junit4] 2> 682972 INFO (coreLoadExecutor-3174-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok c:control_collection x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
[junit4] 2> 682973 INFO (coreLoadExecutor-3174-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok c:control_collection x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 682973 INFO (coreLoadExecutor-3174-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok c:control_collection x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/control-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/control-001/cores/collection1/data/]
[junit4] 2> 682973 INFO (coreLoadExecutor-3174-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok c:control_collection x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa2421
[junit4] 2> 683001 WARN (coreLoadExecutor-3174-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok c:control_collection x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 683053 INFO (coreLoadExecutor-3174-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok c:control_collection x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 683053 INFO (coreLoadExecutor-3174-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok c:control_collection x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 683054 INFO (coreLoadExecutor-3174-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok c:control_collection x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 683054 INFO (coreLoadExecutor-3174-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok c:control_collection x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 683056 INFO (coreLoadExecutor-3174-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok c:control_collection x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1e7fa4[collection1] main]
[junit4] 2> 683057 INFO (coreLoadExecutor-3174-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 683066 INFO (coreLoadExecutor-3174-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 683066 INFO (coreLoadExecutor-3174-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok c:control_collection x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 683072 INFO (searcherExecutor-3175-thread-1-processing-n:127.0.0.1:40734_ok x:collection1 c:control_collection) [n:127.0.0.1:40734_ok c:control_collection x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1e7fa4[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 683073 INFO (coreLoadExecutor-3174-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok c:control_collection x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559763925420998656
[junit4] 2> 683079 INFO (coreZkRegister-3167-thread-1-processing-n:127.0.0.1:40734_ok x:collection1 c:control_collection) [n:127.0.0.1:40734_ok c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 683079 INFO (coreZkRegister-3167-thread-1-processing-n:127.0.0.1:40734_ok x:collection1 c:control_collection) [n:127.0.0.1:40734_ok c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 683079 INFO (coreZkRegister-3167-thread-1-processing-n:127.0.0.1:40734_ok x:collection1 c:control_collection) [n:127.0.0.1:40734_ok c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:40734/ok/collection1/
[junit4] 2> 683079 INFO (coreZkRegister-3167-thread-1-processing-n:127.0.0.1:40734_ok x:collection1 c:control_collection) [n:127.0.0.1:40734_ok c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 683079 INFO (coreZkRegister-3167-thread-1-processing-n:127.0.0.1:40734_ok x:collection1 c:control_collection) [n:127.0.0.1:40734_ok c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:40734/ok/collection1/ has no replicas
[junit4] 2> 683083 INFO (coreZkRegister-3167-thread-1-processing-n:127.0.0.1:40734_ok x:collection1 c:control_collection) [n:127.0.0.1:40734_ok c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:40734/ok/collection1/ shard1
[junit4] 2> 683166 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 683166 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37981/solr ready
[junit4] 2> 683167 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 683234 INFO (coreZkRegister-3167-thread-1-processing-n:127.0.0.1:40734_ok x:collection1 c:control_collection) [n:127.0.0.1:40734_ok c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 683250 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-1-001/cores/collection1
[junit4] 2> 683251 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-1-001
[junit4] 2> 683253 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 683254 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@84d64f{/ok,null,AVAILABLE}
[junit4] 2> 683255 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@527a99{SSL,[ssl, http/1.1]}{127.0.0.1:35036}
[junit4] 2> 683255 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.Server Started @684578ms
[junit4] 2> 683255 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/ok, hostPort=35036, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-1-001/cores}
[junit4] 2> 683256 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 683256 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.5.0
[junit4] 2> 683256 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 683256 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 683256 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-19T12:19:55.514Z
[junit4] 2> 683259 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 683259 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-1-001/solr.xml
[junit4] 2> 683265 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 683266 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37981/solr
[junit4] 2> 683273 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:35036_ok ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 683275 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:35036_ok ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 683276 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:35036_ok ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35036_ok
[junit4] 2> 683277 INFO (zkCallback-3645-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 683278 INFO (zkCallback-3650-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 683279 INFO (zkCallback-3641-thread-2-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 683809 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:35036_ok ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-1-001/cores
[junit4] 2> 683809 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:35036_ok ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 683811 INFO (OverseerStateUpdate-97485245215735813-127.0.0.1:40734_ok-n_0000000000) [n:127.0.0.1:40734_ok ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 684839 INFO (coreLoadExecutor-3185-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
[junit4] 2> 684854 INFO (coreLoadExecutor-3185-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
[junit4] 2> 684858 INFO (coreLoadExecutor-3185-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
[junit4] 2> 684859 INFO (coreLoadExecutor-3185-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 684859 INFO (coreLoadExecutor-3185-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-1-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-1-001/cores/collection1/data/]
[junit4] 2> 684859 INFO (coreLoadExecutor-3185-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa2421
[junit4] 2> 684868 WARN (coreLoadExecutor-3185-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 684924 INFO (coreLoadExecutor-3185-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 684924 INFO (coreLoadExecutor-3185-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 684926 INFO (coreLoadExecutor-3185-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 684926 INFO (coreLoadExecutor-3185-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 684927 INFO (coreLoadExecutor-3185-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@ba6b81[collection1] main]
[junit4] 2> 684928 INFO (coreLoadExecutor-3185-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 684929 INFO (coreLoadExecutor-3185-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 684929 INFO (coreLoadExecutor-3185-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 684932 INFO (searcherExecutor-3186-thread-1-processing-n:127.0.0.1:35036_ok x:collection1 c:collection1) [n:127.0.0.1:35036_ok c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@ba6b81[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 684933 INFO (coreLoadExecutor-3185-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559763927371350016
[junit4] 2> 684940 INFO (coreZkRegister-3180-thread-1-processing-n:127.0.0.1:35036_ok x:collection1 c:collection1) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 684940 INFO (coreZkRegister-3180-thread-1-processing-n:127.0.0.1:35036_ok x:collection1 c:collection1) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 684940 INFO (coreZkRegister-3180-thread-1-processing-n:127.0.0.1:35036_ok x:collection1 c:collection1) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:35036/ok/collection1/
[junit4] 2> 684940 INFO (coreZkRegister-3180-thread-1-processing-n:127.0.0.1:35036_ok x:collection1 c:collection1) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 684940 INFO (coreZkRegister-3180-thread-1-processing-n:127.0.0.1:35036_ok x:collection1 c:collection1) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:35036/ok/collection1/ has no replicas
[junit4] 2> 684943 INFO (coreZkRegister-3180-thread-1-processing-n:127.0.0.1:35036_ok x:collection1 c:collection1) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:35036/ok/collection1/ shard1
[junit4] 2> 685094 INFO (coreZkRegister-3180-thread-1-processing-n:127.0.0.1:35036_ok x:collection1 c:collection1) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 685150 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-2-001/cores/collection1
[junit4] 2> 685151 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-2-001
[junit4] 2> 685153 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 685154 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@123114a{/ok,null,AVAILABLE}
[junit4] 2> 685155 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@139fc61{SSL,[ssl, http/1.1]}{127.0.0.1:38945}
[junit4] 2> 685156 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.Server Started @686478ms
[junit4] 2> 685156 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/ok, hostPort=38945, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-2-001/cores}
[junit4] 2> 685156 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 685157 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.5.0
[junit4] 2> 685157 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 685157 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 685157 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-19T12:19:57.415Z
[junit4] 2> 685159 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 685159 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-2-001/solr.xml
[junit4] 2> 685166 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 685167 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37981/solr
[junit4] 2> 685175 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:38945_ok ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 685177 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:38945_ok ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 685179 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:38945_ok ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38945_ok
[junit4] 2> 685180 INFO (zkCallback-3641-thread-1-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 685180 INFO (zkCallback-3645-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 685181 INFO (zkCallback-3650-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 685181 INFO (zkCallback-3656-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 685430 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:38945_ok ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-2-001/cores
[junit4] 2> 685430 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:38945_ok ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 685431 INFO (OverseerStateUpdate-97485245215735813-127.0.0.1:40734_ok-n_0000000000) [n:127.0.0.1:40734_ok ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 686461 INFO (coreLoadExecutor-3196-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
[junit4] 2> 686483 INFO (coreLoadExecutor-3196-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
[junit4] 2> 686490 INFO (coreLoadExecutor-3196-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
[junit4] 2> 686492 INFO (coreLoadExecutor-3196-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 686492 INFO (coreLoadExecutor-3196-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-2-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-2-001/cores/collection1/data/]
[junit4] 2> 686492 INFO (coreLoadExecutor-3196-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa2421
[junit4] 2> 686505 WARN (coreLoadExecutor-3196-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 686569 INFO (coreLoadExecutor-3196-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 686569 INFO (coreLoadExecutor-3196-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 686570 INFO (coreLoadExecutor-3196-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 686570 INFO (coreLoadExecutor-3196-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 686572 INFO (coreLoadExecutor-3196-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@380150[collection1] main]
[junit4] 2> 686573 INFO (coreLoadExecutor-3196-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 686574 INFO (coreLoadExecutor-3196-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 686574 INFO (coreLoadExecutor-3196-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 686579 INFO (searcherExecutor-3197-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 c:collection1) [n:127.0.0.1:38945_ok c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@380150[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 686579 INFO (coreLoadExecutor-3196-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559763929097306112
[junit4] 2> 686585 INFO (coreZkRegister-3191-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 c:collection1) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
[junit4] 2> 686585 INFO (updateExecutor-3653-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 686585 INFO (recoveryExecutor-3654-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 686586 INFO (recoveryExecutor-3654-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 686586 INFO (recoveryExecutor-3654-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
[junit4] 2> 686586 INFO (recoveryExecutor-3654-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 686586 INFO (recoveryExecutor-3654-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [https://127.0.0.1:35036/ok/collection1/] and I am [https://127.0.0.1:38945/ok/collection1/]
[junit4] 2> 686589 INFO (recoveryExecutor-3654-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [https://127.0.0.1:35036/ok]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:38945_ok&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 686661 INFO (qtp14541142-14026) [n:127.0.0.1:35036_ok ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 686662 INFO (qtp14541142-14026) [n:127.0.0.1:35036_ok ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
[junit4] 2> 686662 INFO (qtp14541142-14026) [n:127.0.0.1:35036_ok ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:38945_ok, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"https://127.0.0.1:38945/ok","node_name":"127.0.0.1:38945_ok","state":"down"}
[junit4] 2> 686735 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-3-001/cores/collection1
[junit4] 2> 686736 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-3-001
[junit4] 2> 686739 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 686740 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@17330c6{/ok,null,AVAILABLE}
[junit4] 2> 686741 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@116dccd{SSL,[ssl, http/1.1]}{127.0.0.1:44436}
[junit4] 2> 686741 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.Server Started @688064ms
[junit4] 2> 686741 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/ok, hostPort=44436, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-3-001/cores}
[junit4] 2> 686742 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 686742 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 6.5.0
[junit4] 2> 686742 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 686742 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 686742 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-19T12:19:59Z
[junit4] 2> 686751 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 686751 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15a565063ed000d, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 686751 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-3-001/solr.xml
[junit4] 2> 686759 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 686760 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37981/solr
[junit4] 2> 686767 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:44436_ok ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 686768 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:44436_ok ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 686770 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:44436_ok ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44436_ok
[junit4] 2> 686771 INFO (zkCallback-3645-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 686771 INFO (zkCallback-3650-thread-1-processing-n:127.0.0.1:35036_ok) [n:127.0.0.1:35036_ok ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 686772 INFO (zkCallback-3656-thread-1-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 686773 INFO (zkCallback-3663-thread-1-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 686773 INFO (zkCallback-3641-thread-3-processing-n:127.0.0.1:40734_ok) [n:127.0.0.1:40734_ok ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 687662 INFO (qtp14541142-14026) [n:127.0.0.1:35036_ok ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:38945_ok, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"https://127.0.0.1:38945/ok","node_name":"127.0.0.1:38945_ok","state":"recovering"}
[junit4] 2> 687662 INFO (qtp14541142-14026) [n:127.0.0.1:35036_ok ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 687662 INFO (qtp14541142-14026) [n:127.0.0.1:35036_ok ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:38945_ok&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1001
[junit4] 2> 687886 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:44436_ok ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-3-001/cores
[junit4] 2> 687886 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [n:127.0.0.1:44436_ok ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 687888 INFO (OverseerStateUpdate-97485245215735813-127.0.0.1:40734_ok-n_0000000000) [n:127.0.0.1:40734_ok ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 688903 INFO (coreLoadExecutor-3207-thread-1-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
[junit4] 2> 688917 INFO (coreLoadExecutor-3207-thread-1-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
[junit4] 2> 688921 INFO (coreLoadExecutor-3207-thread-1-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
[junit4] 2> 688922 INFO (coreLoadExecutor-3207-thread-1-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 688923 INFO (coreLoadExecutor-3207-thread-1-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-3-001/cores/collection1], dataDir=[/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001/shard-3-001/cores/collection1/data/]
[junit4] 2> 688923 INFO (coreLoadExecutor-3207-thread-1-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa2421
[junit4] 2> 688931 WARN (coreLoadExecutor-3207-thread-1-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 688994 INFO (coreLoadExecutor-3207-thread-1-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 688994 INFO (coreLoadExecutor-3207-thread-1-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 688995 INFO (coreLoadExecutor-3207-thread-1-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 688995 INFO (coreLoadExecutor-3207-thread-1-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 688996 INFO (coreLoadExecutor-3207-thread-1-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@130bbfc[collection1] main]
[junit4] 2> 688997 INFO (coreLoadExecutor-3207-thread-1-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 688997 INFO (coreLoadExecutor-3207-thread-1-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 688997 INFO (coreLoadExecutor-3207-thread-1-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 689000 INFO (searcherExecutor-3208-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 c:collection1) [n:127.0.0.1:44436_ok c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@130bbfc[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 689001 INFO (coreLoadExecutor-3207-thread-1-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559763931636957184
[junit4] 2> 689005 INFO (coreZkRegister-3202-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 c:collection1) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
[junit4] 2> 689005 INFO (updateExecutor-3660-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 689005 INFO (recoveryExecutor-3661-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 689006 INFO (recoveryExecutor-3661-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 689007 INFO (recoveryExecutor-3661-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
[junit4] 2> 689007 INFO (recoveryExecutor-3661-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 689007 INFO (recoveryExecutor-3661-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [https://127.0.0.1:35036/ok/collection1/] and I am [https://127.0.0.1:44436/ok/collection1/]
[junit4] 2> 689009 INFO (recoveryExecutor-3661-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [https://127.0.0.1:35036/ok]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:44436_ok&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 689015 INFO (qtp14541142-14025) [n:127.0.0.1:35036_ok ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 689016 INFO (qtp14541142-14025) [n:127.0.0.1:35036_ok ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
[junit4] 2> 689016 INFO (qtp14541142-14025) [n:127.0.0.1:35036_ok ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:44436_ok, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"https://127.0.0.1:44436/ok","node_name":"127.0.0.1:44436_ok","state":"down"}
[junit4] 2> 689097 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.SolrTestCaseJ4 ###Starting test
[junit4] 2> 689097 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):330
[junit4] 1> -
[junit4] 1> replica:core_node1 rstate:active live:true
[junit4] 1> replica:core_node2 rstate:recovering live:true
[junit4] 1> replica:core_node3 rstate:down live:true
[junit4] 2> 690016 INFO (qtp14541142-14025) [n:127.0.0.1:35036_ok ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:44436_ok, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"https://127.0.0.1:44436/ok","node_name":"127.0.0.1:44436_ok","state":"recovering"}
[junit4] 2> 690016 INFO (qtp14541142-14025) [n:127.0.0.1:35036_ok ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 690016 INFO (qtp14541142-14025) [n:127.0.0.1:35036_ok ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:44436_ok&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node3&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1001
[junit4] 1> -
[junit4] 1> replica:core_node1 rstate:active live:true
[junit4] 1> replica:core_node2 rstate:recovering live:true
[junit4] 1> replica:core_node3 rstate:recovering live:true
[junit4] 1> -
[junit4] 1> replica:core_node1 rstate:active live:true
[junit4] 1> replica:core_node2 rstate:recovering live:true
[junit4] 1> replica:core_node3 rstate:recovering live:true
[junit4] 1> -
[junit4] 1> replica:core_node1 rstate:active live:true
[junit4] 1> replica:core_node2 rstate:recovering live:true
[junit4] 1> replica:core_node3 rstate:recovering live:true
[junit4] 1> -
[junit4] 1> replica:core_node1 rstate:active live:true
[junit4] 1> replica:core_node2 rstate:recovering live:true
[junit4] 1> replica:core_node3 rstate:recovering live:true
[junit4] 1> -
[junit4] 1> replica:core_node1 rstate:active live:true
[junit4] 1> replica:core_node2 rstate:recovering live:true
[junit4] 1> replica:core_node3 rstate:recovering live:true
[junit4] 2> 694664 INFO (recoveryExecutor-3654-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [https://127.0.0.1:35036/ok/collection1/] - recoveringAfterStartup=[true]
[junit4] 2> 694664 INFO (recoveryExecutor-3654-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=https://127.0.0.1:38945/ok START replicas=[https://127.0.0.1:35036/ok/collection1/] nUpdates=100
[junit4] 2> 694673 INFO (qtp14541142-14024) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 694674 INFO (qtp14541142-14024) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1] webapp=/ok path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 694675 INFO (recoveryExecutor-3654-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 694675 INFO (recoveryExecutor-3654-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
[junit4] 2> 694675 INFO (recoveryExecutor-3654-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 694675 INFO (recoveryExecutor-3654-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 694676 INFO (recoveryExecutor-3654-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 694676 INFO (recoveryExecutor-3654-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
[junit4] 2> 694676 INFO (recoveryExecutor-3654-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
[junit4] 2> 694676 INFO (recoveryExecutor-3654-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
[junit4] 2> 694676 INFO (recoveryExecutor-3654-thread-1-processing-n:127.0.0.1:38945_ok x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
[junit4] 1> -
[junit4] 1> replica:core_node1 rstate:active live:true
[junit4] 1> replica:core_node2 rstate:active live:true
[junit4] 1> replica:core_node3 rstate:recovering live:true
[junit4] 1> -
[junit4] 1> replica:core_node1 rstate:active live:true
[junit4] 1> replica:core_node2 rstate:active live:true
[junit4] 1> replica:core_node3 rstate:recovering live:true
[junit4] 2> 697022 INFO (recoveryExecutor-3661-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [https://127.0.0.1:35036/ok/collection1/] - recoveringAfterStartup=[true]
[junit4] 2> 697022 INFO (recoveryExecutor-3661-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=https://127.0.0.1:44436/ok START replicas=[https://127.0.0.1:35036/ok/collection1/] nUpdates=100
[junit4] 2> 697029 INFO (qtp14541142-14022) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 697029 INFO (qtp14541142-14022) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1] webapp=/ok path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 697031 INFO (recoveryExecutor-3661-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:1.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 697031 INFO (recoveryExecutor-3661-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
[junit4] 2> 697031 INFO (recoveryExecutor-3661-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 697031 INFO (recoveryExecutor-3661-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 697032 INFO (recoveryExecutor-3661-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 697032 INFO (recoveryExecutor-3661-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
[junit4] 2> 697032 INFO (recoveryExecutor-3661-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
[junit4] 2> 697032 INFO (recoveryExecutor-3661-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
[junit4] 2> 697032 INFO (recoveryExecutor-3661-thread-1-processing-n:127.0.0.1:44436_ok x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
[junit4] 1> -
[junit4] 1> replica:core_node1 rstate:active live:true
[junit4] 1> replica:core_node2 rstate:active live:true
[junit4] 1> replica:core_node3 rstate:active live:true
[junit4] 1> no one is recoverying
[junit4] 2> 697099 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
[junit4] 2> 697164 INFO (qtp15928925-14081) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.S.Request [collection1] webapp=/ok path=/schema/fields/inplace_updatable_int params={wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 697226 INFO (qtp23365279-14047) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.S.Request [collection1] webapp=/ok path=/schema/fields/inplace_updatable_float params={wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 697230 INFO (qtp15928925-14087) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.S.Request [collection1] webapp=/ok path=/schema/fields/_version_ params={wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 697231 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4] 2> 697232 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@182cd52[collection1] realtime]
[junit4] 2> 697232 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:* (-9223372036854775807)} 0 1
[junit4] 2> 697298 INFO (qtp19330434-13984) [n:127.0.0.1:40734_ok c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 697298 INFO (qtp19330434-13984) [n:127.0.0.1:40734_ok c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 697299 INFO (qtp19330434-13984) [n:127.0.0.1:40734_ok c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 697299 INFO (qtp19330434-13984) [n:127.0.0.1:40734_ok c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/ok path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 1
[junit4] 2> 697314 INFO (qtp15928925-14087) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 697315 INFO (qtp15928925-14087) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 697315 INFO (qtp15928925-14087) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 697315 INFO (qtp23365279-14048) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 697315 INFO (qtp15928925-14087) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/ok path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:35036/ok/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 2
[junit4] 2> 697316 INFO (qtp23365279-14048) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 697317 INFO (qtp23365279-14048) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 697317 INFO (qtp23365279-14048) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/ok path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:35036/ok/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 2
[junit4] 2> 697317 INFO (qtp14541142-14022) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 697317 INFO (qtp14541142-14022) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 697318 INFO (qtp14541142-14022) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 697318 INFO (qtp14541142-14022) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/ok path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:35036/ok/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 1
[junit4] 2> 697319 INFO (qtp14541142-14019) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/ok path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 15
[junit4] 2> 697328 INFO (qtp23365279-14048) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/ok path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:35036/ok/collection1/&wt=javabin&version=2}{add=[0 (2001)]} 0 2
[junit4] 2> 697428 INFO (qtp23365279-14049) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1fd8e16[collection1] realtime]
[junit4] 2> 697430 INFO (qtp23365279-14049) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/ok path=/update params={update.distrib=FROMLEADER&_version_=2003&distrib.from=https://127.0.0.1:35036/okcollection1/&wt=javabin&version=2}{deleteByQuery=inplace_updatable_float:101.0 (-2003)} 0 5
[junit4] 2> 697525 INFO (qtp23365279-14054) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 Reordered DBQs detected. Update=add{_version_=2002,id=0} DBQs=[DBQ{version=2003,q=inplace_updatable_float:101.0}]
[junit4] 2> 697537 INFO (qtp23365279-14054) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1db141e[collection1] realtime]
[junit4] 2> 697538 INFO (qtp23365279-14054) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@f0be80[collection1] realtime]
[junit4] 2> 697539 INFO (qtp23365279-14054) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/ok path=/update params={update.distrib=FROMLEADER&distrib.inplace.prevversion=2001&distrib.from=https://127.0.0.1:35036/ok/collection1/&wt=javabin&version=2&distrib.inplace.update=true}{add=[0 (2002)]} 0 14
[junit4] 2> 697546 INFO (qtp14541142-14021) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 697547 INFO (qtp14541142-14021) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 697548 INFO (qtp14541142-14021) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 697548 INFO (qtp14541142-14021) [n:127.0.0.1:35036_ok c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]
[...truncated too long message...]
] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:38945_ok
[junit4] 2> 1038172 INFO (zkCallback-3663-thread-3-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 1038172 INFO (zkCallback-3663-thread-4-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:44436_ok
[junit4] 2> 1038173 INFO (zkCallback-3663-thread-2-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1038173 INFO (zkCallback-3663-thread-4-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok ] o.a.s.c.Overseer Overseer (id=97485245215735823-127.0.0.1:44436_ok-n_0000000003) starting
[junit4] 2> 1039167 WARN (zkCallback-3656-thread-3-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy Closed, skipping sync up.
[junit4] 2> 1039167 INFO (zkCallback-3656-thread-3-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SolrCore [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@15504fd
[junit4] 2> 1039672 WARN (zkCallback-3656-thread-2-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 1040126 INFO (zkCallback-3656-thread-3-processing-n:127.0.0.1:38945_ok) [n:127.0.0.1:38945_ok c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
[junit4] 2> 1040129 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 1040131 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@139fc61{SSL,[ssl, http/1.1]}{127.0.0.1:0}
[junit4] 2> 1040132 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@123114a{/ok,null,UNAVAILABLE}
[junit4] 2> 1040132 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.ChaosMonkey monkey: stop shard! 44436
[junit4] 2> 1040132 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=21746242
[junit4] 2> 1040135 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.Overseer Overseer (id=97485245215735823-127.0.0.1:44436_ok-n_0000000003) closing
[junit4] 2> 1040135 INFO (OverseerStateUpdate-97485245215735823-127.0.0.1:44436_ok-n_0000000003) [n:127.0.0.1:44436_ok ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:44436_ok
[junit4] 2> 1040135 WARN (zkCallback-3663-thread-4-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 1040673 WARN (zkCallback-3663-thread-2-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.SyncStrategy Closed, skipping sync up.
[junit4] 2> 1040674 INFO (zkCallback-3663-thread-2-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.SolrCore [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@d503d0
[junit4] 2> 1041235 INFO (zkCallback-3663-thread-2-processing-n:127.0.0.1:44436_ok) [n:127.0.0.1:44436_ok c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
[junit4] 2> 1041238 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 1041239 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@116dccd{SSL,[ssl, http/1.1]}{127.0.0.1:0}
[junit4] 2> 1041239 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@17330c6{/ok,null,UNAVAILABLE}
[junit4] 2> 1041242 INFO (TEST-TestInPlaceUpdatesDistrib.test-seed#[9A6C0179F16FF134]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37981 37981
[junit4] 2> 1046254 INFO (Thread-1387) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37981 37981
[junit4] 2> 1046255 WARN (Thread-1387) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/aliases.json
[junit4] 2> 4 /solr/security.json
[junit4] 2> 4 /solr/configs/conf1
[junit4] 2> 3 /solr/collections/collection1/state.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/clusterstate.json
[junit4] 2> 5 /solr/clusterprops.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 358 /solr/overseer/collection-queue-work
[junit4] 2> 24 /solr/overseer/queue
[junit4] 2> 7 /solr/overseer/queue-work
[junit4] 2> 5 /solr/live_nodes
[junit4] 2> 5 /solr/collections
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestInPlaceUpdatesDistrib -Dtests.method=test -Dtests.seed=9A6C0179F16FF134 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es-AR -Dtests.timezone=America/Indiana/Knox -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 365s J2 | TestInPlaceUpdatesDistrib.test <<<
[junit4] > Throwable #1: java.lang.AssertionError: Thread pool didn't terminate within 10 secs
[junit4] > at __randomizedtesting.SeedInfo.seed([9A6C0179F16FF134:12383EA35F939CCC]:0)
[junit4] > at org.apache.solr.update.TestInPlaceUpdatesDistrib.reorderedDBQsWithInPlaceUpdatesShouldNotThrowReplicaInLIRTest(TestInPlaceUpdatesDistrib.java:740)
[junit4] > at org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:145)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1046271 INFO (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> 1046271 INFO (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=16460351
[junit4] 2> 1046272 INFO (coreCloseExecutor-3235-thread-1) [ x:collection1] o.a.s.c.SolrCore [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@b93290
[junit4] 2> 1046303 INFO (coreCloseExecutor-3235-thread-1) [ x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
[junit4] 2> 1046307 INFO (SUITE-TestInPlaceUpdatesDistrib-seed#[9A6C0179F16FF134]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_9A6C0179F16FF134-001
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=false,coord=crazy): {}, locale=es-AR, timezone=America/Indiana/Knox
[junit4] 2> NOTE: Linux 4.4.0-53-generic i386/Oracle Corporation 1.8.0_121 (32-bit)/cpus=12,threads=1,free=116116608,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [TestCloudRecovery, TestStressReorder, WrapperMergePolicyFactoryTest, AddBlockUpdateTest, TestUniqueKeyFieldResource, HdfsChaosMonkeySafeLeaderTest, PKIAuthenticationIntegrationTest, HdfsDirectoryFactoryTest, TestCollationField, ShowFileRequestHandlerTest, HdfsRecoverLeaseTest, SortSpecParsingTest, DistributedFacetPivotSmallAdvancedTest, ResponseLogComponentTest, TestMaxScoreQueryParser, SmileWriterTest, TestManagedSchemaAPI, HighlighterMaxOffsetTest, TestCloudSchemaless, ShufflingReplicaListTransformerTest, TestInitParams, ConfigSetsAPITest, DistributedFacetExistsSmallTest, TestSolrConfigHandlerCloud, ParsingFieldUpdateProcessorsTest, CoreAdminHandlerTest, TestRuleBasedAuthorizationPlugin, TestSort, TestRandomFlRTGCloud, TestImpersonationWithHadoopAuth, SecurityConfHandlerTest, UnloadDistributedZkTest, TestDocBasedVersionConstraints, StatsComponentTest, DistribJoinFromCollectionTest, TestSchemaResource, CloneFieldUpdateProcessorFactoryTest, TestConfigSets, TestCryptoKeys, TestSSLRandomization, TestSizeLimitedDistributedMap, ManagedSchemaRoundRobinCloudTest, DOMUtilTest, TestSolrFieldCacheMBean, TestIntervalFaceting, TestPhraseSuggestions, RequestLoggingTest, TestJsonRequest, MissingSegmentRecoveryTest, ResponseHeaderTest, CircularListTest, CollectionReloadTest, TestSchemaManager, TestReversedWildcardFilterFactory, TestCustomSort, BasicDistributedZk2Test, JSONWriterTest, CursorMarkTest, TestTolerantSearch, TestGroupingSearch, RollingRestartTest, SimpleFacetsTest, TestStressLiveNodes, SuggesterTest, SolrCoreCheckLockOnStartupTest, TestInPlaceUpdatesDistrib]
[junit4] Completed [280/696 (1!)] on J2 in 365.37s, 1 test, 1 failure <<< FAILURES!
[...truncated 56224 lines...]