You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2017/02/10 09:54:49 UTC
[JENKINS] Lucene-Solr-Tests-master - Build # 1653 - Still Failing
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/1653/
1 tests failed.
FAILED: org.apache.solr.cloud.MissingSegmentRecoveryTest.testLeaderRecovery
Error Message:
expected:<10> but was:<0>
Stack Trace:
java.lang.AssertionError: expected:<10> but was:<0>
at __randomizedtesting.SeedInfo.seed([AF6B90BC08563644:FF3E08BF51778059]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.MissingSegmentRecoveryTest.testLeaderRecovery(MissingSegmentRecoveryTest.java:108)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 12131 lines...]
[junit4] Suite: org.apache.solr.cloud.MissingSegmentRecoveryTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.MissingSegmentRecoveryTest_AF6B90BC08563644-001/init-core-data-001
[junit4] 2> 739592 INFO (SUITE-MissingSegmentRecoveryTest-seed#[AF6B90BC08563644]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields
[junit4] 2> 739593 INFO (SUITE-MissingSegmentRecoveryTest-seed#[AF6B90BC08563644]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 739593 INFO (SUITE-MissingSegmentRecoveryTest-seed#[AF6B90BC08563644]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.MissingSegmentRecoveryTest_AF6B90BC08563644-001/tempDir-001
[junit4] 2> 739593 INFO (SUITE-MissingSegmentRecoveryTest-seed#[AF6B90BC08563644]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 739597 INFO (Thread-1320) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 739597 INFO (Thread-1320) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 739697 INFO (SUITE-MissingSegmentRecoveryTest-seed#[AF6B90BC08563644]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:49226
[junit4] 2> 739709 INFO (jetty-launcher-1685-thread-1) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 739711 INFO (jetty-launcher-1685-thread-2) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 739720 INFO (jetty-launcher-1685-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6d27c1ba{/solr,null,AVAILABLE}
[junit4] 2> 739722 INFO (jetty-launcher-1685-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4bbd8688{SSL,[ssl, http/1.1]}{127.0.0.1:34519}
[junit4] 2> 739722 INFO (jetty-launcher-1685-thread-1) [ ] o.e.j.s.Server Started @743425ms
[junit4] 2> 739722 INFO (jetty-launcher-1685-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34519}
[junit4] 2> 739723 ERROR (jetty-launcher-1685-thread-1) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 739730 INFO (jetty-launcher-1685-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 739730 INFO (jetty-launcher-1685-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 739730 INFO (jetty-launcher-1685-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 739730 INFO (jetty-launcher-1685-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-10T09:21:20.235Z
[junit4] 2> 739729 INFO (jetty-launcher-1685-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6a092305{/solr,null,AVAILABLE}
[junit4] 2> 739732 INFO (jetty-launcher-1685-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@626ac073{SSL,[ssl, http/1.1]}{127.0.0.1:39783}
[junit4] 2> 739732 INFO (jetty-launcher-1685-thread-2) [ ] o.e.j.s.Server Started @743434ms
[junit4] 2> 739732 INFO (jetty-launcher-1685-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=39783}
[junit4] 2> 739732 ERROR (jetty-launcher-1685-thread-2) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 739732 INFO (jetty-launcher-1685-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 739732 INFO (jetty-launcher-1685-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 739732 INFO (jetty-launcher-1685-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 739732 INFO (jetty-launcher-1685-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-10T09:21:20.237Z
[junit4] 2> 739753 INFO (jetty-launcher-1685-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 739753 INFO (jetty-launcher-1685-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 739777 INFO (jetty-launcher-1685-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49226/solr
[junit4] 2> 739789 INFO (jetty-launcher-1685-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49226/solr
[junit4] 2> 739809 INFO (jetty-launcher-1685-thread-2) [n:127.0.0.1:39783_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:39783_solr
[junit4] 2> 739810 INFO (jetty-launcher-1685-thread-2) [n:127.0.0.1:39783_solr ] o.a.s.c.Overseer Overseer (id=97433582312030213-127.0.0.1:39783_solr-n_0000000000) starting
[junit4] 2> 739822 INFO (jetty-launcher-1685-thread-2) [n:127.0.0.1:39783_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39783_solr
[junit4] 2> 739824 INFO (zkCallback-1696-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 739824 INFO (zkCallback-1695-thread-1-processing-n:127.0.0.1:39783_solr) [n:127.0.0.1:39783_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 739829 INFO (jetty-launcher-1685-thread-1) [n:127.0.0.1:34519_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34519_solr
[junit4] 2> 739830 INFO (zkCallback-1695-thread-1-processing-n:127.0.0.1:39783_solr) [n:127.0.0.1:39783_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 739836 INFO (zkCallback-1696-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 739907 INFO (jetty-launcher-1685-thread-1) [n:127.0.0.1:34519_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.MissingSegmentRecoveryTest_AF6B90BC08563644-001/tempDir-001/node1/.
[junit4] 2> 739910 INFO (jetty-launcher-1685-thread-2) [n:127.0.0.1:39783_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.MissingSegmentRecoveryTest_AF6B90BC08563644-001/tempDir-001/node2/.
[junit4] 2> 740146 INFO (SUITE-MissingSegmentRecoveryTest-seed#[AF6B90BC08563644]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 740146 INFO (SUITE-MissingSegmentRecoveryTest-seed#[AF6B90BC08563644]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:49226/solr ready
[junit4] 2> 740158 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testLeaderRecovery
[junit4] 2> 740178 INFO (qtp1548116953-7373) [n:127.0.0.1:39783_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=MissingSegmentRecoveryTest&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 740179 INFO (OverseerThreadFactory-2902-thread-1-processing-n:127.0.0.1:39783_solr) [n:127.0.0.1:39783_solr ] o.a.s.c.CreateCollectionCmd Create collection MissingSegmentRecoveryTest
[junit4] 2> 740294 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=MissingSegmentRecoveryTest_shard1_replica2&action=CREATE&numShards=1&collection=MissingSegmentRecoveryTest&shard=shard1&wt=javabin&version=2
[junit4] 2> 740319 INFO (qtp1909158773-7365) [n:127.0.0.1:34519_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=MissingSegmentRecoveryTest_shard1_replica1&action=CREATE&numShards=1&collection=MissingSegmentRecoveryTest&shard=shard1&wt=javabin&version=2
[junit4] 2> 740422 INFO (zkCallback-1696-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 740422 INFO (zkCallback-1695-thread-3-processing-n:127.0.0.1:39783_solr) [n:127.0.0.1:39783_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 741312 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 741318 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.s.IndexSchema [MissingSegmentRecoveryTest_shard1_replica2] Schema name=minimal
[junit4] 2> 741320 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 741325 INFO (qtp1909158773-7365) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 741325 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'MissingSegmentRecoveryTest_shard1_replica2' using configuration from collection MissingSegmentRecoveryTest
[junit4] 2> 741326 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrCore [[MissingSegmentRecoveryTest_shard1_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.MissingSegmentRecoveryTest_AF6B90BC08563644-001/tempDir-001/node2/MissingSegmentRecoveryTest_shard1_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.MissingSegmentRecoveryTest_AF6B90BC08563644-001/tempDir-001/node2/./MissingSegmentRecoveryTest_shard1_replica2/data/]
[junit4] 2> 741331 INFO (qtp1909158773-7365) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.s.IndexSchema [MissingSegmentRecoveryTest_shard1_replica1] Schema name=minimal
[junit4] 2> 741333 INFO (qtp1909158773-7365) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 741333 INFO (qtp1909158773-7365) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'MissingSegmentRecoveryTest_shard1_replica1' using configuration from collection MissingSegmentRecoveryTest
[junit4] 2> 741334 INFO (qtp1909158773-7365) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SolrCore [[MissingSegmentRecoveryTest_shard1_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.MissingSegmentRecoveryTest_AF6B90BC08563644-001/tempDir-001/node1/MissingSegmentRecoveryTest_shard1_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.MissingSegmentRecoveryTest_AF6B90BC08563644-001/tempDir-001/node1/./MissingSegmentRecoveryTest_shard1_replica1/data/]
[junit4] 2> 741377 INFO (qtp1909158773-7365) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 741377 INFO (qtp1909158773-7365) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 741377 INFO (qtp1909158773-7365) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 741377 INFO (qtp1909158773-7365) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 741378 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 741378 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 741379 INFO (qtp1909158773-7365) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@797caea4[MissingSegmentRecoveryTest_shard1_replica1] main]
[junit4] 2> 741379 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 741379 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 741380 INFO (qtp1909158773-7365) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 741380 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@4c47ede9[MissingSegmentRecoveryTest_shard1_replica2] main]
[junit4] 2> 741380 INFO (qtp1909158773-7365) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 741380 INFO (qtp1909158773-7365) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 741380 INFO (qtp1909158773-7365) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1558937318861045760
[junit4] 2> 741381 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 741381 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 741381 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 741381 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1558937318862094336
[junit4] 2> 741381 INFO (searcherExecutor-2908-thread-1-processing-n:127.0.0.1:34519_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica1] Registered new searcher Searcher@797caea4[MissingSegmentRecoveryTest_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 741382 INFO (searcherExecutor-2907-thread-1-processing-n:127.0.0.1:39783_solr x:MissingSegmentRecoveryTest_shard1_replica2 s:shard1 c:MissingSegmentRecoveryTest) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica2] Registered new searcher Searcher@4c47ede9[MissingSegmentRecoveryTest_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 741386 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 741386 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 741386 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:39783/solr/MissingSegmentRecoveryTest_shard1_replica2/
[junit4] 2> 741387 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.PeerSync PeerSync: core=MissingSegmentRecoveryTest_shard1_replica2 url=https://127.0.0.1:39783/solr START replicas=[https://127.0.0.1:34519/solr/MissingSegmentRecoveryTest_shard1_replica1/] nUpdates=100
[junit4] 2> 741387 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.PeerSync PeerSync: core=MissingSegmentRecoveryTest_shard1_replica2 url=https://127.0.0.1:39783/solr DONE. We have no versions. sync failed.
[junit4] 2> 741391 INFO (qtp1909158773-7360) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.S.Request [MissingSegmentRecoveryTest_shard1_replica1] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 741391 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 741391 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 741394 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:39783/solr/MissingSegmentRecoveryTest_shard1_replica2/ shard1
[junit4] 2> 741497 INFO (zkCallback-1696-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 741497 INFO (zkCallback-1695-thread-2-processing-n:127.0.0.1:39783_solr) [n:127.0.0.1:39783_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 741545 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 741550 INFO (qtp1548116953-7375) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=MissingSegmentRecoveryTest_shard1_replica2&action=CREATE&numShards=1&collection=MissingSegmentRecoveryTest&shard=shard1&wt=javabin&version=2} status=0 QTime=1255
[junit4] 2> 741649 INFO (zkCallback-1696-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 741649 INFO (zkCallback-1695-thread-2-processing-n:127.0.0.1:39783_solr) [n:127.0.0.1:39783_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 742389 INFO (qtp1909158773-7365) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=MissingSegmentRecoveryTest_shard1_replica1&action=CREATE&numShards=1&collection=MissingSegmentRecoveryTest&shard=shard1&wt=javabin&version=2} status=0 QTime=2070
[junit4] 2> 742393 INFO (qtp1548116953-7373) [n:127.0.0.1:39783_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 742492 INFO (zkCallback-1696-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 742492 INFO (zkCallback-1695-thread-2-processing-n:127.0.0.1:39783_solr) [n:127.0.0.1:39783_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 743394 INFO (qtp1548116953-7373) [n:127.0.0.1:39783_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=MissingSegmentRecoveryTest&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=3215
[junit4] 2> 743402 INFO (qtp1909158773-7363) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [MissingSegmentRecoveryTest_shard1_replica1] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:39783/solr/MissingSegmentRecoveryTest_shard1_replica2/&wt=javabin&version=2}{add=[0 (1558937320977072128), 1 (1558937320979169280), 2 (1558937320979169281), 3 (1558937320979169282), 4 (1558937320979169283), 5 (1558937320979169284), 6 (1558937320979169285), 7 (1558937320979169286), 8 (1558937320979169287), 9 (1558937320979169288)]} 0 1
[junit4] 2> 743402 INFO (qtp1548116953-7374) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [MissingSegmentRecoveryTest_shard1_replica2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[0 (1558937320977072128), 1 (1558937320979169280), 2 (1558937320979169281), 3 (1558937320979169282), 4 (1558937320979169283), 5 (1558937320979169284), 6 (1558937320979169285), 7 (1558937320979169286), 8 (1558937320979169287), 9 (1558937320979169288)]} 0 3
[junit4] 2> 743410 INFO (qtp1548116953-7359) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 743410 INFO (qtp1548116953-7359) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@5688663d
[junit4] 2> 743415 INFO (qtp1909158773-7364) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 743415 INFO (qtp1909158773-7364) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@632b07ae
[junit4] 2> 743443 INFO (qtp1548116953-7359) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@58be0f84[MissingSegmentRecoveryTest_shard1_replica2] main]
[junit4] 2> 743444 INFO (qtp1548116953-7359) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 743444 INFO (searcherExecutor-2907-thread-1-processing-n:127.0.0.1:39783_solr x:MissingSegmentRecoveryTest_shard1_replica2 s:shard1 c:MissingSegmentRecoveryTest r:core_node1) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica2] Registered new searcher Searcher@58be0f84[MissingSegmentRecoveryTest_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):C10)))}
[junit4] 2> 743444 INFO (qtp1548116953-7359) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [MissingSegmentRecoveryTest_shard1_replica2] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:39783/solr/MissingSegmentRecoveryTest_shard1_replica2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 34
[junit4] 2> 743445 INFO (qtp1909158773-7364) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1d1700ea[MissingSegmentRecoveryTest_shard1_replica1] main]
[junit4] 2> 743445 INFO (searcherExecutor-2908-thread-1-processing-n:127.0.0.1:34519_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest r:core_node2) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica1] Registered new searcher Searcher@1d1700ea[MissingSegmentRecoveryTest_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):C10)))}
[junit4] 2> 743445 INFO (qtp1909158773-7364) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 743445 INFO (qtp1909158773-7364) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [MissingSegmentRecoveryTest_shard1_replica1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:39783/solr/MissingSegmentRecoveryTest_shard1_replica2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 30
[junit4] 2> 743446 INFO (qtp1548116953-7371) [n:127.0.0.1:39783_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node1 x:MissingSegmentRecoveryTest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [MissingSegmentRecoveryTest_shard1_replica2] webapp=/solr path=/update params={_stateVer_=MissingSegmentRecoveryTest:4&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 43
[junit4] 2> 743448 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@4bbd8688{SSL,[ssl, http/1.1]}{127.0.0.1:0}
[junit4] 2> 743449 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1663788799
[junit4] 2> 743450 INFO (coreCloseExecutor-2917-thread-1) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica1] CLOSING SolrCore org.apache.solr.core.SolrCore@7317e6e0
[junit4] 2> 743451 INFO (coreCloseExecutor-2917-thread-1) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.MissingSegmentRecoveryTest.shard1.replica1
[junit4] 2> 743452 WARN (zkCallback-1696-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 743452 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 743452 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15a2753aee70006, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 743453 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6d27c1ba{/solr,null,UNAVAILABLE}
[junit4] 2> 743453 INFO (zkCallback-1702-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 743453 INFO (zkCallback-1695-thread-3-processing-n:127.0.0.1:39783_solr) [n:127.0.0.1:39783_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 743459 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 743459 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4615dc2a{/solr,null,AVAILABLE}
[junit4] 2> 743461 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@288d503c{SSL,[ssl, http/1.1]}{127.0.0.1:34519}
[junit4] 2> 743461 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.e.j.s.Server Started @747164ms
[junit4] 2> 743461 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34519}
[junit4] 2> 743462 ERROR (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 743462 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 743462 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 743462 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 743462 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-10T09:21:23.967Z
[junit4] 2> 743464 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 743482 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49226/solr
[junit4] 2> 743489 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [n:127.0.0.1:34519_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 743491 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [n:127.0.0.1:34519_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34519_solr
[junit4] 2> 743493 INFO (zkCallback-1702-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 743493 INFO (zkCallback-1695-thread-2-processing-n:127.0.0.1:39783_solr) [n:127.0.0.1:39783_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 743497 INFO (zkCallback-1707-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 743547 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [n:127.0.0.1:34519_solr ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.MissingSegmentRecoveryTest_AF6B90BC08563644-001/tempDir-001/node1/.
[junit4] 2> 743547 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [n:127.0.0.1:34519_solr ] o.a.s.c.CorePropertiesLocator Cores are: [MissingSegmentRecoveryTest_shard1_replica1]
[junit4] 2> 743555 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 743562 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.s.IndexSchema [MissingSegmentRecoveryTest_shard1_replica1] Schema name=minimal
[junit4] 2> 743564 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 743565 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'MissingSegmentRecoveryTest_shard1_replica1' using configuration from collection MissingSegmentRecoveryTest
[junit4] 2> 743565 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SolrCore [[MissingSegmentRecoveryTest_shard1_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.MissingSegmentRecoveryTest_AF6B90BC08563644-001/tempDir-001/node1/MissingSegmentRecoveryTest_shard1_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.MissingSegmentRecoveryTest_AF6B90BC08563644-001/tempDir-001/node1/./MissingSegmentRecoveryTest_shard1_replica1/data/]
[junit4] 2> 743597 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 743597 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 743598 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 743598 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 743599 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica1] CLOSING SolrCore org.apache.solr.core.SolrCore@1fc56287
[junit4] 2> 743599 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.MissingSegmentRecoveryTest.shard1.replica1
[junit4] 2> 743601 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.CoreContainer Found active leader, will attempt to create fresh core and recover.
[junit4] 2> 743601 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SolrCore Updating index properties... index=index.20170210002124106
[junit4] 2> 743604 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SolrCore [[MissingSegmentRecoveryTest_shard1_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.MissingSegmentRecoveryTest_AF6B90BC08563644-001/tempDir-001/node1/MissingSegmentRecoveryTest_shard1_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.MissingSegmentRecoveryTest_AF6B90BC08563644-001/tempDir-001/node1/./MissingSegmentRecoveryTest_shard1_replica1/data/]
[junit4] 2> 743613 INFO (OldIndexDirectoryCleanupThreadForCore-MissingSegmentRecoveryTest_shard1_replica1) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.DirectoryFactory Found 1 old index directories to clean-up under /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.MissingSegmentRecoveryTest_AF6B90BC08563644-001/tempDir-001/node1/./MissingSegmentRecoveryTest_shard1_replica1/data/
[junit4] 2> 743618 INFO (OldIndexDirectoryCleanupThreadForCore-MissingSegmentRecoveryTest_shard1_replica1) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.DirectoryFactory Deleted old index directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.MissingSegmentRecoveryTest_AF6B90BC08563644-001/tempDir-001/node1/./MissingSegmentRecoveryTest_shard1_replica1/data/index
[junit4] 2> 743634 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 743634 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 743635 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 743635 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 743637 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@144d10ef[MissingSegmentRecoveryTest_shard1_replica1] main]
[junit4] 2> 743638 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 743639 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 743639 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 743639 INFO (coreLoadExecutor-2924-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 743641 INFO (searcherExecutor-2930-thread-1-processing-n:127.0.0.1:34519_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest r:core_node2) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica1] Registered new searcher Searcher@144d10ef[MissingSegmentRecoveryTest_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 743646 INFO (coreZkRegister-2919-thread-1-processing-n:127.0.0.1:34519_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest r:core_node2) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.ZkController Core needs to recover:MissingSegmentRecoveryTest_shard1_replica1
[junit4] 2> 743646 INFO (updateExecutor-1704-thread-1-processing-n:127.0.0.1:34519_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest r:core_node2) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 743646 INFO (recoveryExecutor-1705-thread-1-processing-n:127.0.0.1:34519_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest r:core_node2) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 743647 INFO (recoveryExecutor-1705-thread-1-processing-n:127.0.0.1:34519_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest r:core_node2) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[1558937320979169288, 1558937320979169287, 1558937320979169286, 1558937320979169285, 1558937320979169284, 1558937320979169283, 1558937320979169282, 1558937320979169281, 1558937320979169280, 1558937320977072128]]
[junit4] 2> 743647 INFO (recoveryExecutor-1705-thread-1-processing-n:127.0.0.1:34519_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest r:core_node2) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[MissingSegmentRecoveryTest_shard1_replica1]
[junit4] 2> 743647 INFO (recoveryExecutor-1705-thread-1-processing-n:127.0.0.1:34519_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest r:core_node2) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 743647 INFO (recoveryExecutor-1705-thread-1-processing-n:127.0.0.1:34519_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest r:core_node2) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.RecoveryStrategy Publishing state of core [MissingSegmentRecoveryTest_shard1_replica1] as recovering, leader is [https://127.0.0.1:39783/solr/MissingSegmentRecoveryTest_shard1_replica2/] and I am [https://127.0.0.1:34519/solr/MissingSegmentRecoveryTest_shard1_replica1/]
[junit4] 2> 743650 INFO (recoveryExecutor-1705-thread-1-processing-n:127.0.0.1:34519_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest r:core_node2) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [https://127.0.0.1:39783/solr]; [WaitForState: action=PREPRECOVERY&core=MissingSegmentRecoveryTest_shard1_replica2&nodeName=127.0.0.1:34519_solr&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 743653 INFO (qtp1548116953-7369) [n:127.0.0.1:39783_solr ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 743654 INFO (qtp1548116953-7369) [n:127.0.0.1:39783_solr ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see MissingSegmentRecoveryTest_shard1_replica2 (shard1 of MissingSegmentRecoveryTest) have state: recovering
[junit4] 2> 743654 INFO (qtp1548116953-7369) [n:127.0.0.1:39783_solr ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=MissingSegmentRecoveryTest, shard=shard1, thisCore=MissingSegmentRecoveryTest_shard1_replica2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=active, localState=active, nodeName=127.0.0.1:34519_solr, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"MissingSegmentRecoveryTest_shard1_replica1","base_url":"https://127.0.0.1:34519/solr","node_name":"127.0.0.1:34519_solr","state":"active"}
[junit4] 2> 743747 INFO (qtp2013759343-7440) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.S.Request [MissingSegmentRecoveryTest_shard1_replica1] webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=MissingSegmentRecoveryTest:4&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=https://127.0.0.1:34519/solr/MissingSegmentRecoveryTest_shard1_replica1/|https://127.0.0.1:39783/solr/MissingSegmentRecoveryTest_shard1_replica2/&rows=10&version=2&q=*:*&NOW=1486718484245&isShard=true&wt=javabin} hits=0 status=0 QTime=0
[junit4] 2> 743748 INFO (qtp2013759343-7438) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.S.Request [MissingSegmentRecoveryTest_shard1_replica1] webapp=/solr path=/select params={q=*:*&_stateVer_=MissingSegmentRecoveryTest:4&wt=javabin&version=2} hits=0 status=0 QTime=7
[junit4] 2> 743748 INFO (qtp1548116953-7372) [n:127.0.0.1:39783_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=MissingSegmentRecoveryTest&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 743749 INFO (zkCallback-1695-thread-2-processing-n:127.0.0.1:39783_solr) [n:127.0.0.1:39783_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 743750 INFO (zkCallback-1707-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 743758 INFO (OverseerCollectionConfigSetProcessor-97433582312030213-127.0.0.1:39783_solr-n_0000000000) [n:127.0.0.1:39783_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 743758 INFO (OverseerThreadFactory-2902-thread-2-processing-n:127.0.0.1:39783_solr) [n:127.0.0.1:39783_solr ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
[junit4] 2> 743759 INFO (qtp1548116953-7359) [n:127.0.0.1:39783_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.MissingSegmentRecoveryTest.shard1.replica2
[junit4] 2> 743759 INFO (qtp1548116953-7359) [n:127.0.0.1:39783_solr ] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica2] CLOSING SolrCore org.apache.solr.core.SolrCore@2b5de560
[junit4] 2> 743761 INFO (qtp1548116953-7359) [n:127.0.0.1:39783_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.MissingSegmentRecoveryTest.shard1.replica2
[junit4] 2> 743765 ERROR (OverseerThreadFactory-2902-thread-2-processing-n:127.0.0.1:39783_solr) [n:127.0.0.1:39783_solr ] o.a.s.c.OverseerCollectionMessageHandler Error from shard: https://127.0.0.1:34519/solr
[junit4] 2> org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:34519/solr
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:641)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
[junit4] 2> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
[junit4] 2> at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:163)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:34519 failed to respond
[junit4] 2> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
[junit4] 2> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
[junit4] 2> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
[junit4] 2> at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
[junit4] 2> at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
[junit4] 2> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
[junit4] 2> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
[junit4] 2> at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:118)
[junit4] 2> at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
[junit4] 2> at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
[junit4] 2> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
[junit4] 2> at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
[junit4] 2> at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
[junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
[junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:532)
[junit4] 2> ... 12 more
[junit4] 2> 743771 INFO (zkCallback-1707-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.ShardLeaderElectionContext There may be a better leader candidate than us - going back into recovery
[junit4] 2> 743771 INFO (zkCallback-1707-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration.
[junit4] 2> 743771 INFO (zkCallback-1707-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.ActionThrottle The last leader attempt started 1ms ago.
[junit4] 2> 743771 INFO (zkCallback-1707-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.ActionThrottle Throttling leader attempts - waiting for 4998ms
[junit4] 2> 743772 INFO (qtp1548116953-7359) [n:127.0.0.1:39783_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=MissingSegmentRecoveryTest_shard1_replica2&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=13
[junit4] 2> 743876 INFO (zkCallback-1707-thread-2-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/MissingSegmentRecoveryTest/state.json] for collection [MissingSegmentRecoveryTest] has occurred - updating... (live nodes size: [2])
[junit4] 2> 744479 INFO (qtp1548116953-7372) [n:127.0.0.1:39783_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=MissingSegmentRecoveryTest&action=DELETE&wt=javabin&version=2} status=0 QTime=730
[junit4] 2> 744480 INFO (TEST-MissingSegmentRecoveryTest.testLeaderRecovery-seed#[AF6B90BC08563644]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testLeaderRecovery
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=MissingSegmentRecoveryTest -Dtests.method=testLeaderRecovery -Dtests.seed=AF6B90BC08563644 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=da-DK -Dtests.timezone=SystemV/YST9YDT -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 4.33s J2 | MissingSegmentRecoveryTest.testLeaderRecovery <<<
[junit4] > Throwable #1: java.lang.AssertionError: expected:<10> but was:<0>
[junit4] > at __randomizedtesting.SeedInfo.seed([AF6B90BC08563644:FF3E08BF51778059]:0)
[junit4] > at org.apache.solr.cloud.MissingSegmentRecoveryTest.testLeaderRecovery(MissingSegmentRecoveryTest.java:108)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 744485 INFO (httpUriRequest-1709-thread-1-processing-x:MissingSegmentRecoveryTest_shard1_replica1 r:core_node2 n:127.0.0.1:34519_solr https:////127.0.0.1:39783//solr s:shard1 c:MissingSegmentRecoveryTest) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.h.i.e.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://127.0.0.1:39783: The target server failed to respond
[junit4] 2> 744485 INFO (httpUriRequest-1709-thread-1-processing-x:MissingSegmentRecoveryTest_shard1_replica1 r:core_node2 n:127.0.0.1:34519_solr https:////127.0.0.1:39783//solr s:shard1 c:MissingSegmentRecoveryTest) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.h.i.e.RetryExec Retrying request to {s}->https://127.0.0.1:39783
[junit4] 2> 744485 INFO (jetty-launcher-1685-thread-1) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@626ac073{SSL,[ssl, http/1.1]}{127.0.0.1:0}
[junit4] 2> 744486 INFO (jetty-launcher-1685-thread-1) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1758844520
[junit4] 2> 744486 INFO (jetty-launcher-1685-thread-2) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@288d503c{SSL,[ssl, http/1.1]}{127.0.0.1:34519}
[junit4] 2> 744487 INFO (jetty-launcher-1685-thread-2) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1596930080
[junit4] 2> 744488 WARN (jetty-launcher-1685-thread-2) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[MissingSegmentRecoveryTest_shard1_replica1] coreNodeName=[core_node2]
[junit4] 2> 744493 ERROR (recoveryExecutor-1705-thread-1-processing-n:127.0.0.1:34519_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest r:core_node2) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.RecoveryStrategy Error while trying to recover. core=MissingSegmentRecoveryTest_shard1_replica1:java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: https://127.0.0.1:39783/solr
[junit4] 2> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
[junit4] 2> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
[junit4] 2> at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:614)
[junit4] 2> at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:591)
[junit4] 2> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:351)
[junit4] 2> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222)
[junit4] 2> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: https://127.0.0.1:39783/solr
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:634)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.lambda$httpUriRequest$0(HttpSolrClient.java:319)
[junit4] 2> ... 5 more
[junit4] 2> Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:39783 [/127.0.0.1] failed: Connection refused
[junit4] 2> at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
[junit4] 2> at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
[junit4] 2> at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
[junit4] 2> at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
[junit4] 2> at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
[junit4] 2> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
[junit4] 2> at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
[junit4] 2> at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
[junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
[junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:532)
[junit4] 2> ... 6 more
[junit4] 2> Caused by: java.net.ConnectException: Connection refused
[junit4] 2> at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] 2> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
[junit4] 2> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
[junit4] 2> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
[junit4] 2> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[junit4] 2> at java.net.Socket.connect(Socket.java:589)
[junit4] 2> at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:337)
[junit4] 2> at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
[junit4] 2> ... 16 more
[junit4] 2>
[junit4] 2> 744493 INFO (recoveryExecutor-1705-thread-1-processing-n:127.0.0.1:34519_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest r:core_node2) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.RecoveryStrategy Replay not started, or was not successful... still buffering updates.
[junit4] 2> 744493 INFO (recoveryExecutor-1705-thread-1-processing-n:127.0.0.1:34519_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest r:core_node2) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.RecoveryStrategy RecoveryStrategy has been closed
[junit4] 2> 744493 INFO (recoveryExecutor-1705-thread-1-processing-n:127.0.0.1:34519_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest r:core_node2) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.RecoveryStrategy Finished recovery process, successful=[false]
[junit4] 2> 744493 WARN (updateExecutor-1704-thread-2-processing-n:127.0.0.1:34519_solr x:MissingSegmentRecoveryTest_shard1_replica1 s:shard1 c:MissingSegmentRecoveryTest r:core_node2) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.u.DefaultSolrCoreState Skipping recovery because Solr is shutdown
[junit4] 2> 744497 INFO (jetty-launcher-1685-thread-1) [ ] o.a.s.c.Overseer Overseer (id=97433582312030213-127.0.0.1:39783_solr-n_0000000000) closing
[junit4] 2> 744497 INFO (OverseerStateUpdate-97433582312030213-127.0.0.1:39783_solr-n_0000000000) [n:127.0.0.1:39783_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:39783_solr
[junit4] 2> 744499 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15a2753aee7000b, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 744654 ERROR (qtp1548116953-7369) [n:127.0.0.1:39783_solr ] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Solr is shutting down
[junit4] 2> at org.apache.solr.handler.admin.PrepRecoveryOp.execute(PrepRecoveryOp.java:183)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:378)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:388)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
[junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:171)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:747)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:728)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:509)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:462)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
[junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
[junit4] 2> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
[junit4] 2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 744655 INFO (qtp1548116953-7369) [n:127.0.0.1:39783_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:34519_solr&onlyIfLeaderActive=true&core=MissingSegmentRecoveryTest_shard1_replica2&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=400 QTime=1001
[junit4] 2> 745999 WARN (zkCallback-1707-thread-2-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 745999 WARN (zkCallback-1695-thread-3-processing-n:127.0.0.1:39783_solr) [n:127.0.0.1:39783_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 746000 INFO (jetty-launcher-1685-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 746000 INFO (jetty-launcher-1685-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6a092305{/solr,null,UNAVAILABLE}
[junit4] 2> 748770 ERROR (zkCallback-1707-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.Overseer Could not create Overseer node
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:391)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:388)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:388)
[junit4] 2> at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:728)
[junit4] 2> at org.apache.solr.cloud.Overseer.getStateUpdateQueue(Overseer.java:601)
[junit4] 2> at org.apache.solr.cloud.Overseer.getStateUpdateQueue(Overseer.java:588)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:310)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:677)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:340)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:268)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 748770 INFO (zkCallback-1707-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica1] CLOSING SolrCore org.apache.solr.core.SolrCore@549afa5f
[junit4] 2> 748771 WARN (zkCallback-1707-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.c.RecoveryStrategy Stopping recovery for core=[MissingSegmentRecoveryTest_shard1_replica1] coreNodeName=[core_node2]
[junit4] 2> 748772 INFO (zkCallback-1707-thread-1-processing-n:127.0.0.1:34519_solr) [n:127.0.0.1:34519_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node2 x:MissingSegmentRecoveryTest_shard1_replica1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.MissingSegmentRecoveryTest.shard1.replica1
[junit4] 2> 748773 INFO (jetty-launcher-1685-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 748774 INFO (jetty-launcher-1685-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4615dc2a{/solr,null,UNAVAILABLE}
[junit4] 2> 748775 INFO (SUITE-MissingSegmentRecoveryTest-seed#[AF6B90BC08563644]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:49226 49226
[junit4] 2> 748863 INFO (Thread-1320) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:49226 49226
[junit4] 2> 748864 WARN (Thread-1320) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/aliases.json
[junit4] 2> 3 /solr/security.json
[junit4] 2> 3 /solr/configs/conf
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/collections/MissingSegmentRecoveryTest/state.json
[junit4] 2> 4 /solr/clusterstate.json
[junit4] 2> 4 /solr/clusterprops.json
[junit4] 2> 2 /solr/overseer_elect/election/97433582312030213-127.0.0.1:39783_solr-n_0000000000
[junit4] 2> 2 /solr/collections/MissingSegmentRecoveryTest/leader_elect/shard1/election/97433582312030213-core_node1-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 16 /solr/overseer/queue
[junit4] 2> 4 /solr/collections
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 3 /solr/overseer/queue-work
[junit4] 2>
[junit4] 2> 748864 INFO (SUITE-MissingSegmentRecoveryTest-seed#[AF6B90BC08563644]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.MissingSegmentRecoveryTest_AF6B90BC08563644-001
[junit4] 2> Feb 10, 2017 9:21:29 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {_version_=Lucene50(blocksize=128), id=PostingsFormat(name=MockRandom)}, docValues:{}, maxPointsInLeafNode=1083, maxMBSortInHeap=5.709560432413866, sim=RandomSimilarity(queryNorm=true): {}, locale=da-DK, timezone=SystemV/YST9YDT
[junit4] 2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=4,threads=1,free=136654008,total=499646464
[junit4] 2> NOTE: All tests run in this JVM: [LeaderElectionContextKeyTest, UpdateParamsTest, CoreAdminHandlerTest, StatelessScriptUpdateProcessorFactoryTest, TestInPlaceUpdatesDistrib, CustomCollectionTest, TestFieldSortValues, SolrCoreCheckLockOnStartupTest, TestBlendedInfixSuggestions, CoreMergeIndexesAdminHandlerTest, CloneFieldUpdateProcessorFactoryTest, OutputWriterTest, TestSolrConfigHandlerCloud, DistributedFacetPivotSmallAdvancedTest, TestTrieFacet, TestTestInjection, TestBinaryField, TestMaxScoreQueryParser, TestNumericTerms32, TestSchemalessBufferedUpdates, TestDocBasedVersionConstraints, TestDistribIDF, TestRandomFlRTGCloud, LeaderElectionTest, SimpleCollectionCreateDeleteTest, TestDocSet, TestReplicaProperties, StatsComponentTest, DocValuesNotIndexedTest, TestLuceneMatchVersion, TestRestoreCore, DirectUpdateHandlerOptimizeTest, HdfsRecoverLeaseTest, DeleteNodeTest, FieldMutatingUpdateProcessorTest, TestZkChroot, HdfsBasicDistributedZk2Test, SuggestComponentTest, BlockJoinFacetDistribTest, TestApiFramework, TestComplexPhraseLeadingWildcard, TestJavabinTupleStreamParser, FieldAnalysisRequestHandlerTest, WrapperMergePolicyFactoryTest, DataDrivenBlockJoinTest, TestCloudManagedSchema, OverseerTest, TestShardHandlerFactory, SubstringBytesRefFilterTest, MergeStrategyTest, DistributedFacetExistsSmallTest, EchoParamsTest, TestUtils, TestSolrXml, TestCollationField, TestSubQueryTransformer, PeerSyncReplicationTest, TestConfig, DistribJoinFromCollectionTest, SolrGraphiteReporterTest, TestSchemaSimilarityResource, OverriddenZkACLAndCredentialsProvidersTest, TestDownShardTolerantSearch, FacetPivotSmallTest, BufferStoreTest, LeaderInitiatedRecoveryOnCommitTest, CdcrReplicationHandlerTest, RequestHandlersTest, TestRawTransformer, TestClusterStateMutator, MissingSegmentRecoveryTest]
[junit4] Completed [222/694 (1!)] on J2 in 10.04s, 1 test, 1 failure <<< FAILURES!
[...truncated 14133 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:775: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:117: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/common-build.xml:1937: Rat problems were found!
Total time: 74 minutes 1 second
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-Tests-master - Build # 1660 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/1660/
1 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.ltr.TestLTROnSolrCloud
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.ltr.TestLTROnSolrCloud: 1) Thread[id=601, name=OverseerHdfsCoreFailoverThread-97446407584874527-127.0.0.1:37157_solr-n_0000000001, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139) at java.lang.Thread.run(Thread.java:745)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.ltr.TestLTROnSolrCloud:
1) Thread[id=601, name=OverseerHdfsCoreFailoverThread-97446407584874527-127.0.0.1:37157_solr-n_0000000001, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
at java.lang.Thread.run(Thread.java:745)
at __randomizedtesting.SeedInfo.seed([9BBB7A8EADEC50B7]:0)
Build Log:
[...truncated 19419 lines...]
[junit4] Suite: org.apache.solr.ltr.TestLTROnSolrCloud
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/init-core-data-001
[junit4] 2> 18969 INFO (SUITE-TestLTROnSolrCloud-seed#[9BBB7A8EADEC50B7]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields
[junit4] 2> 18973 INFO (SUITE-TestLTROnSolrCloud-seed#[9BBB7A8EADEC50B7]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 18978 INFO (TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[9BBB7A8EADEC50B7]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testSimpleQuery
[junit4] 2> 18982 INFO (TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[9BBB7A8EADEC50B7]) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 12 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001
[junit4] 2> 18988 INFO (TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[9BBB7A8EADEC50B7]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 18989 INFO (Thread-45) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 18990 INFO (Thread-45) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 19089 INFO (TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[9BBB7A8EADEC50B7]) [ ] o.a.s.c.ZkTestServer start zk server on port:47704
[junit4] 2> 19173 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15a32fdccbc0000, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 19194 INFO (jetty-launcher-17-thread-11) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 19196 INFO (jetty-launcher-17-thread-2) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 19198 INFO (jetty-launcher-17-thread-3) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 19199 INFO (jetty-launcher-17-thread-4) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 19200 INFO (jetty-launcher-17-thread-5) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 19201 INFO (jetty-launcher-17-thread-1) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 19202 INFO (jetty-launcher-17-thread-6) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 19203 INFO (jetty-launcher-17-thread-7) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 19206 INFO (jetty-launcher-17-thread-8) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 19207 INFO (jetty-launcher-17-thread-9) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 19209 INFO (jetty-launcher-17-thread-10) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 19210 INFO (jetty-launcher-17-thread-12) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 19230 INFO (jetty-launcher-17-thread-12) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@60b3f77b{/solr,null,AVAILABLE}
[junit4] 2> 19232 INFO (jetty-launcher-17-thread-9) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6efee63b{/solr,null,AVAILABLE}
[junit4] 2> 19243 INFO (jetty-launcher-17-thread-8) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@80b389b{/solr,null,AVAILABLE}
[junit4] 2> 19251 INFO (jetty-launcher-17-thread-7) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@507a7af4{/solr,null,AVAILABLE}
[junit4] 2> 19258 INFO (jetty-launcher-17-thread-6) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4e1a7b05{/solr,null,AVAILABLE}
[junit4] 2> 19260 INFO (jetty-launcher-17-thread-5) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@12d1d912{/solr,null,AVAILABLE}
[junit4] 2> 19264 INFO (jetty-launcher-17-thread-4) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4cf44373{/solr,null,AVAILABLE}
[junit4] 2> 19284 INFO (jetty-launcher-17-thread-3) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@45ccef3b{/solr,null,AVAILABLE}
[junit4] 2> 19291 INFO (jetty-launcher-17-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@31abd86b{/solr,null,AVAILABLE}
[junit4] 2> 19301 INFO (jetty-launcher-17-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5f8fc541{/solr,null,AVAILABLE}
[junit4] 2> 19321 INFO (jetty-launcher-17-thread-6) [ ] o.e.j.s.AbstractConnector Started ServerConnector@5c45c540{SSL,[ssl, http/1.1]}{127.0.0.1:49586}
[junit4] 2> 19321 INFO (jetty-launcher-17-thread-6) [ ] o.e.j.s.Server Started @21689ms
[junit4] 2> 19321 INFO (jetty-launcher-17-thread-6) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=49586}
[junit4] 2> 19321 INFO (jetty-launcher-17-thread-3) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4174dd95{SSL,[ssl, http/1.1]}{127.0.0.1:33056}
[junit4] 2> 19321 INFO (jetty-launcher-17-thread-3) [ ] o.e.j.s.Server Started @21689ms
[junit4] 2> 19321 INFO (jetty-launcher-17-thread-3) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33056}
[junit4] 2> 19322 INFO (jetty-launcher-17-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@9217da8{SSL,[ssl, http/1.1]}{127.0.0.1:50305}
[junit4] 2> 19322 INFO (jetty-launcher-17-thread-2) [ ] o.e.j.s.Server Started @21690ms
[junit4] 2> 19322 INFO (jetty-launcher-17-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50305}
[junit4] 2> 19323 INFO (jetty-launcher-17-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@253c4d65{SSL,[ssl, http/1.1]}{127.0.0.1:51138}
[junit4] 2> 19323 INFO (jetty-launcher-17-thread-1) [ ] o.e.j.s.Server Started @21691ms
[junit4] 2> 19323 INFO (jetty-launcher-17-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=51138}
[junit4] 2> 19323 INFO (jetty-launcher-17-thread-4) [ ] o.e.j.s.AbstractConnector Started ServerConnector@19b5d680{SSL,[ssl, http/1.1]}{127.0.0.1:37157}
[junit4] 2> 19323 INFO (jetty-launcher-17-thread-4) [ ] o.e.j.s.Server Started @21691ms
[junit4] 2> 19323 INFO (jetty-launcher-17-thread-4) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37157}
[junit4] 2> 19324 INFO (jetty-launcher-17-thread-5) [ ] o.e.j.s.AbstractConnector Started ServerConnector@2ff37792{SSL,[ssl, http/1.1]}{127.0.0.1:57247}
[junit4] 2> 19324 INFO (jetty-launcher-17-thread-5) [ ] o.e.j.s.Server Started @21692ms
[junit4] 2> 19324 INFO (jetty-launcher-17-thread-5) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=57247}
[junit4] 2> 19324 INFO (jetty-launcher-17-thread-7) [ ] o.e.j.s.AbstractConnector Started ServerConnector@22ffd0d1{SSL,[ssl, http/1.1]}{127.0.0.1:33627}
[junit4] 2> 19324 INFO (jetty-launcher-17-thread-7) [ ] o.e.j.s.Server Started @21692ms
[junit4] 2> 19324 INFO (jetty-launcher-17-thread-7) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33627}
[junit4] 2> 19325 INFO (jetty-launcher-17-thread-8) [ ] o.e.j.s.AbstractConnector Started ServerConnector@391bb644{SSL,[ssl, http/1.1]}{127.0.0.1:59736}
[junit4] 2> 19325 INFO (jetty-launcher-17-thread-8) [ ] o.e.j.s.Server Started @21693ms
[junit4] 2> 19325 INFO (jetty-launcher-17-thread-8) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59736}
[junit4] 2> 19325 INFO (jetty-launcher-17-thread-12) [ ] o.e.j.s.AbstractConnector Started ServerConnector@30c50d45{SSL,[ssl, http/1.1]}{127.0.0.1:50231}
[junit4] 2> 19327 INFO (jetty-launcher-17-thread-9) [ ] o.e.j.s.AbstractConnector Started ServerConnector@547889b1{SSL,[ssl, http/1.1]}{127.0.0.1:48750}
[junit4] 2> 19328 INFO (jetty-launcher-17-thread-9) [ ] o.e.j.s.Server Started @21695ms
[junit4] 2> 19328 INFO (jetty-launcher-17-thread-9) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=48750}
[junit4] 2> 19332 INFO (jetty-launcher-17-thread-12) [ ] o.e.j.s.Server Started @21694ms
[junit4] 2> 19332 INFO (jetty-launcher-17-thread-12) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50231}
[junit4] 2> 19348 INFO (jetty-launcher-17-thread-11) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@d703f26{/solr,null,AVAILABLE}
[junit4] 2> 19349 INFO (jetty-launcher-17-thread-11) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4e560109{SSL,[ssl, http/1.1]}{127.0.0.1:55621}
[junit4] 2> 19349 INFO (jetty-launcher-17-thread-11) [ ] o.e.j.s.Server Started @21717ms
[junit4] 2> 19349 INFO (jetty-launcher-17-thread-11) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55621}
[junit4] 2> 19321 ERROR (jetty-launcher-17-thread-6) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 19322 ERROR (jetty-launcher-17-thread-3) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 19322 ERROR (jetty-launcher-17-thread-2) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 19354 INFO (jetty-launcher-17-thread-10) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4f0e5ad5{/solr,null,AVAILABLE}
[junit4] 2> 19323 ERROR (jetty-launcher-17-thread-1) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 19323 ERROR (jetty-launcher-17-thread-4) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 19324 ERROR (jetty-launcher-17-thread-5) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 19354 INFO (jetty-launcher-17-thread-10) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4f28add5{SSL,[ssl, http/1.1]}{127.0.0.1:46551}
[junit4] 2> 19325 ERROR (jetty-launcher-17-thread-7) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 19354 INFO (jetty-launcher-17-thread-10) [ ] o.e.j.s.Server Started @21722ms
[junit4] 2> 19325 ERROR (jetty-launcher-17-thread-8) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 19354 INFO (jetty-launcher-17-thread-10) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46551}
[junit4] 2> 19332 ERROR (jetty-launcher-17-thread-9) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 19332 ERROR (jetty-launcher-17-thread-12) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 19349 ERROR (jetty-launcher-17-thread-11) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 19356 ERROR (jetty-launcher-17-thread-10) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 19356 INFO (jetty-launcher-17-thread-10) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 19356 INFO (jetty-launcher-17-thread-10) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 19356 INFO (jetty-launcher-17-thread-10) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 19356 INFO (jetty-launcher-17-thread-10) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-12T15:42:58.550Z
[junit4] 2> 19354 INFO (jetty-launcher-17-thread-3) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 19357 INFO (jetty-launcher-17-thread-3) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 19357 INFO (jetty-launcher-17-thread-3) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 19357 INFO (jetty-launcher-17-thread-3) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-12T15:42:58.551Z
[junit4] 2> 19354 INFO (jetty-launcher-17-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 19357 INFO (jetty-launcher-17-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 19357 INFO (jetty-launcher-17-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 19358 INFO (jetty-launcher-17-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-12T15:42:58.552Z
[junit4] 2> 19354 INFO (jetty-launcher-17-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 19358 INFO (jetty-launcher-17-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 19358 INFO (jetty-launcher-17-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 19358 INFO (jetty-launcher-17-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-12T15:42:58.552Z
[junit4] 2> 19354 INFO (jetty-launcher-17-thread-4) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 19359 INFO (jetty-launcher-17-thread-4) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 19359 INFO (jetty-launcher-17-thread-4) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 19359 INFO (jetty-launcher-17-thread-4) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-12T15:42:58.553Z
[junit4] 2> 19354 INFO (jetty-launcher-17-thread-5) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 19365 INFO (jetty-launcher-17-thread-5) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 19365 INFO (jetty-launcher-17-thread-5) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 19365 INFO (jetty-launcher-17-thread-5) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-12T15:42:58.559Z
[junit4] 2> 19354 INFO (jetty-launcher-17-thread-7) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 19366 INFO (jetty-launcher-17-thread-7) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 19366 INFO (jetty-launcher-17-thread-7) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 19366 INFO (jetty-launcher-17-thread-7) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-12T15:42:58.560Z
[junit4] 2> 19354 INFO (jetty-launcher-17-thread-8) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 19366 INFO (jetty-launcher-17-thread-8) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 19366 INFO (jetty-launcher-17-thread-8) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 19366 INFO (jetty-launcher-17-thread-8) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-12T15:42:58.560Z
[junit4] 2> 19354 INFO (jetty-launcher-17-thread-9) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 19367 INFO (jetty-launcher-17-thread-9) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 19367 INFO (jetty-launcher-17-thread-9) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 19367 INFO (jetty-launcher-17-thread-9) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-12T15:42:58.561Z
[junit4] 2> 19355 INFO (jetty-launcher-17-thread-12) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 19355 INFO (jetty-launcher-17-thread-11) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 19367 INFO (jetty-launcher-17-thread-12) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 19367 INFO (jetty-launcher-17-thread-11) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 19367 INFO (jetty-launcher-17-thread-12) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 19367 INFO (jetty-launcher-17-thread-11) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 19367 INFO (jetty-launcher-17-thread-12) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-12T15:42:58.561Z
[junit4] 2> 19367 INFO (jetty-launcher-17-thread-11) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-12T15:42:58.561Z
[junit4] 2> 19353 INFO (jetty-launcher-17-thread-6) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 19368 INFO (jetty-launcher-17-thread-6) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 19368 INFO (jetty-launcher-17-thread-6) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 19368 INFO (jetty-launcher-17-thread-6) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-12T15:42:58.562Z
[junit4] 2> 19499 INFO (jetty-launcher-17-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 19500 INFO (jetty-launcher-17-thread-3) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 19501 INFO (jetty-launcher-17-thread-5) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 19501 INFO (jetty-launcher-17-thread-11) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 19501 INFO (jetty-launcher-17-thread-10) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 19501 INFO (jetty-launcher-17-thread-9) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 19512 INFO (jetty-launcher-17-thread-12) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 19512 INFO (jetty-launcher-17-thread-6) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 19550 INFO (jetty-launcher-17-thread-11) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47704/solr
[junit4] 2> 19560 INFO (jetty-launcher-17-thread-9) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47704/solr
[junit4] 2> 19572 INFO (jetty-launcher-17-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47704/solr
[junit4] 2> 19564 INFO (jetty-launcher-17-thread-7) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 19563 INFO (jetty-launcher-17-thread-4) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 19616 INFO (jetty-launcher-17-thread-3) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47704/solr
[junit4] 2> 19616 INFO (jetty-launcher-17-thread-6) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47704/solr
[junit4] 2> 19639 INFO (jetty-launcher-17-thread-12) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47704/solr
[junit4] 2> 19654 INFO (jetty-launcher-17-thread-5) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47704/solr
[junit4] 2> 19573 INFO (jetty-launcher-17-thread-8) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 19573 INFO (jetty-launcher-17-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 19656 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15a32fdccbc000d, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 19679 INFO (jetty-launcher-17-thread-4) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47704/solr
[junit4] 2> 19680 INFO (jetty-launcher-17-thread-10) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47704/solr
[junit4] 2> 19686 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15a32fdccbc0008, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 19711 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15a32fdccbc000e, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 19713 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15a32fdccbc0010, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 19714 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15a32fdccbc0012, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 19734 INFO (jetty-launcher-17-thread-7) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47704/solr
[junit4] 2> 19745 INFO (jetty-launcher-17-thread-8) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47704/solr
[junit4] 2> 19753 INFO (jetty-launcher-17-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47704/solr
[junit4] 2> 19760 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15a32fdccbc0014, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 19763 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15a32fdccbc0016, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 19769 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15a32fdccbc0018, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 19769 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15a32fdccbc001c, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 19793 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15a32fdccbc0020, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 19957 INFO (jetty-launcher-17-thread-5) [n:127.0.0.1:57247_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:57247_solr
[junit4] 2> 19978 INFO (jetty-launcher-17-thread-7) [n:127.0.0.1:33627_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33627_solr
[junit4] 2> 19978 INFO (jetty-launcher-17-thread-8) [n:127.0.0.1:59736_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59736_solr
[junit4] 2> 19978 INFO (jetty-launcher-17-thread-2) [n:127.0.0.1:50305_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50305_solr
[junit4] 2> 19979 INFO (jetty-launcher-17-thread-9) [n:127.0.0.1:48750_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48750_solr
[junit4] 2> 19979 INFO (jetty-launcher-17-thread-4) [n:127.0.0.1:37157_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37157_solr
[junit4] 2> 19980 INFO (jetty-launcher-17-thread-11) [n:127.0.0.1:55621_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55621_solr
[junit4] 2> 19980 INFO (jetty-launcher-17-thread-12) [n:127.0.0.1:50231_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50231_solr
[junit4] 2> 19981 INFO (jetty-launcher-17-thread-10) [n:127.0.0.1:46551_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46551_solr
[junit4] 2> 19982 INFO (jetty-launcher-17-thread-5) [n:127.0.0.1:57247_solr ] o.a.s.c.Overseer Overseer (id=97446407584874525-127.0.0.1:57247_solr-n_0000000000) starting
[junit4] 2> 19983 INFO (jetty-launcher-17-thread-1) [n:127.0.0.1:51138_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51138_solr
[junit4] 2> 20002 INFO (zkCallback-57-thread-1-processing-n:127.0.0.1:50305_solr) [n:127.0.0.1:50305_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
[junit4] 2> 20002 INFO (zkCallback-75-thread-1-processing-n:127.0.0.1:46551_solr) [n:127.0.0.1:46551_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
[junit4] 2> 20017 INFO (zkCallback-76-thread-1-processing-n:127.0.0.1:33627_solr) [n:127.0.0.1:33627_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
[junit4] 2> 20019 INFO (zkCallback-78-thread-1-processing-n:127.0.0.1:51138_solr) [n:127.0.0.1:51138_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
[junit4] 2> 20019 INFO (zkCallback-64-thread-1-processing-n:127.0.0.1:50231_solr) [n:127.0.0.1:50231_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
[junit4] 2> 20184 INFO (zkCallback-75-thread-1-processing-n:127.0.0.1:46551_solr) [n:127.0.0.1:46551_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
[junit4] 2> 20188 INFO (zkCallback-78-thread-1-processing-n:127.0.0.1:51138_solr) [n:127.0.0.1:51138_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
[junit4] 2> 20188 INFO (zkCallback-64-thread-1-processing-n:127.0.0.1:50231_solr) [n:127.0.0.1:50231_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
[junit4] 2> 20188 INFO (zkCallback-76-thread-1-processing-n:127.0.0.1:33627_solr) [n:127.0.0.1:33627_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
[junit4] 2> 20188 INFO (zkCallback-57-thread-1-processing-n:127.0.0.1:50305_solr) [n:127.0.0.1:50305_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (8)
[junit4] 2> 20200 INFO (zkCallback-57-thread-1-processing-n:127.0.0.1:50305_solr) [n:127.0.0.1:50305_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
[junit4] 2> 20200 INFO (zkCallback-75-thread-1-processing-n:127.0.0.1:46551_solr) [n:127.0.0.1:46551_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
[junit4] 2> 20200 INFO (zkCallback-64-thread-1-processing-n:127.0.0.1:50231_solr) [n:127.0.0.1:50231_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
[junit4] 2> 20330 INFO (zkCallback-62-thread-1-processing-n:127.0.0.1:33056_solr) [n:127.0.0.1:33056_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (8)
[junit4] 2> 20330 INFO (zkCallback-74-thread-1-processing-n:127.0.0.1:37157_solr) [n:127.0.0.1:37157_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
[junit4] 2> 20330 INFO (zkCallback-58-thread-1-processing-n:127.0.0.1:55621_solr) [n:127.0.0.1:55621_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
[junit4] 2> 20330 INFO (zkCallback-77-thread-1-processing-n:127.0.0.1:59736_solr) [n:127.0.0.1:59736_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (8)
[junit4] 2> 20330 INFO (zkCallback-63-thread-1-processing-n:127.0.0.1:48750_solr) [n:127.0.0.1:48750_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (9)
[junit4] 2> 20330 INFO (zkCallback-72-thread-1-processing-n:127.0.0.1:57247_solr) [n:127.0.0.1:57247_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (8)
[junit4] 2> 20330 INFO (zkCallback-66-thread-1-processing-n:127.0.0.1:49586_solr) [n:127.0.0.1:49586_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (7)
[junit4] 2> 20331 INFO (zkCallback-72-thread-2-processing-n:127.0.0.1:57247_solr) [n:127.0.0.1:57247_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
[junit4] 2> 20331 INFO (zkCallback-66-thread-2-processing-n:127.0.0.1:49586_solr) [n:127.0.0.1:49586_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
[junit4] 2> 20331 INFO (zkCallback-77-thread-2-processing-n:127.0.0.1:59736_solr) [n:127.0.0.1:59736_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
[junit4] 2> 20331 INFO (zkCallback-76-thread-2-processing-n:127.0.0.1:33627_solr) [n:127.0.0.1:33627_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
[junit4] 2> 20331 INFO (zkCallback-78-thread-2-processing-n:127.0.0.1:51138_solr) [n:127.0.0.1:51138_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
[junit4] 2> 20336 INFO (jetty-launcher-17-thread-3) [n:127.0.0.1:33056_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33056_solr
[junit4] 2> 20392 INFO (zkCallback-74-thread-2-processing-n:127.0.0.1:37157_solr) [n:127.0.0.1:37157_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
[junit4] 2> 20393 INFO (zkCallback-62-thread-3-processing-n:127.0.0.1:33056_solr) [n:127.0.0.1:33056_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (8) -> (9)
[junit4] 2> 20393 INFO (jetty-launcher-17-thread-6) [n:127.0.0.1:49586_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49586_solr
[junit4] 2> 20393 INFO (zkCallback-58-thread-2-processing-n:127.0.0.1:55621_solr) [n:127.0.0.1:55621_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (7) -> (9)
[junit4] 2> 20395 INFO (zkCallback-75-thread-1-processing-n:127.0.0.1:46551_solr) [n:127.0.0.1:46551_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
[junit4] 2> 20396 INFO (zkCallback-63-thread-2-processing-n:127.0.0.1:48750_solr) [n:127.0.0.1:48750_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
[junit4] 2> 20396 INFO (zkCallback-76-thread-2-processing-n:127.0.0.1:33627_solr) [n:127.0.0.1:33627_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
[junit4] 2> 20396 INFO (zkCallback-57-thread-1-processing-n:127.0.0.1:50305_solr) [n:127.0.0.1:50305_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
[junit4] 2> 20396 INFO (zkCallback-64-thread-1-processing-n:127.0.0.1:50231_solr) [n:127.0.0.1:50231_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
[junit4] 2> 20396 INFO (zkCallback-77-thread-2-processing-n:127.0.0.1:59736_solr) [n:127.0.0.1:59736_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
[junit4] 2> 20396 INFO (zkCallback-78-thread-2-processing-n:127.0.0.1:51138_solr) [n:127.0.0.1:51138_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
[junit4] 2> 20398 INFO (zkCallback-72-thread-2-processing-n:127.0.0.1:57247_solr) [n:127.0.0.1:57247_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (11)
[junit4] 2> 20400 INFO (zkCallback-66-thread-2-processing-n:127.0.0.1:49586_solr) [n:127.0.0.1:49586_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (11)
[junit4] 2> 20401 INFO (zkCallback-62-thread-3-processing-n:127.0.0.1:33056_solr) [n:127.0.0.1:33056_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
[junit4] 2> 20401 INFO (zkCallback-74-thread-2-processing-n:127.0.0.1:37157_solr) [n:127.0.0.1:37157_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
[junit4] 2> 20401 INFO (zkCallback-58-thread-2-processing-n:127.0.0.1:55621_solr) [n:127.0.0.1:55621_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (9) -> (10)
[junit4] 2> 20408 INFO (zkCallback-57-thread-1-processing-n:127.0.0.1:50305_solr) [n:127.0.0.1:50305_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (10) -> (11)
[junit4] 2> 20408 INFO (zkCallback-63-thread-2-processing-n:127.0.0.1:48750_solr) [n:127.0.0.1:48750_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (10) -> (11)
[junit4] 2> 20409 INFO (zkCallback-64-thread-1-processing-n:127.0.0.1:50231_solr) [n:127.0.0.1:50231_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (10) -> (11)
[junit4] 2> 20409 INFO (zkCallback-77-thread-2-processing-n:127.0.0.1:59736_solr) [n:127.0.0.1:59736_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (10) -> (11)
[junit4] 2> 20409 INFO (zkCallback-76-thread-2-processing-n:127.0.0.1:33627_solr) [n:127.0.0.1:33627_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (10) -> (11)
[junit4] 2> 20409 INFO (zkCallback-78-thread-2-processing-n:127.0.0.1:51138_solr) [n:127.0.0.1:51138_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (10) -> (11)
[junit4] 2> 20422 INFO (zkCallback-74-thread-1-processing-n:127.0.0.1:37157_solr) [n:127.0.0.1:37157_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (10) -> (11)
[junit4] 2> 20423 INFO (zkCallback-58-thread-1-processing-n:127.0.0.1:55621_solr) [n:127.0.0.1:55621_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (10) -> (11)
[junit4] 2> 20423 INFO (zkCallback-75-thread-1-processing-n:127.0.0.1:46551_solr) [n:127.0.0.1:46551_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (10) -> (11)
[junit4] 2> 20423 INFO (zkCallback-62-thread-2-processing-n:127.0.0.1:33056_solr) [n:127.0.0.1:33056_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (10) -> (11)
[junit4] 2> 20718 INFO (jetty-launcher-17-thread-2) [n:127.0.0.1:50305_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node2/.
[junit4] 2> 20731 INFO (jetty-launcher-17-thread-8) [n:127.0.0.1:59736_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node8/.
[junit4] 2> 20743 INFO (jetty-launcher-17-thread-7) [n:127.0.0.1:33627_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node7/.
[junit4] 2> 20745 INFO (jetty-launcher-17-thread-11) [n:127.0.0.1:55621_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node11/.
[junit4] 2> 20755 INFO (jetty-launcher-17-thread-1) [n:127.0.0.1:51138_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node1/.
[junit4] 2> 20780 INFO (jetty-launcher-17-thread-12) [n:127.0.0.1:50231_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node12/.
[junit4] 2> 20877 INFO (jetty-launcher-17-thread-10) [n:127.0.0.1:46551_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node10/.
[junit4] 2> 21047 INFO (jetty-launcher-17-thread-5) [n:127.0.0.1:57247_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57247_solr
[junit4] 2> 21052 INFO (zkCallback-62-thread-2-processing-n:127.0.0.1:33056_solr) [n:127.0.0.1:33056_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (11) -> (12)
[junit4] 2> 21052 INFO (zkCallback-63-thread-2-processing-n:127.0.0.1:48750_solr) [n:127.0.0.1:48750_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (11) -> (12)
[junit4] 2> 21052 INFO (zkCallback-74-thread-1-processing-n:127.0.0.1:37157_solr) [n:127.0.0.1:37157_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (11) -> (12)
[junit4] 2> 21052 INFO (zkCallback-76-thread-2-processing-n:127.0.0.1:33627_solr) [n:127.0.0.1:33627_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (11) -> (12)
[junit4] 2> 21052 INFO (zkCallback-75-thread-1-processing-n:127.0.0.1:46551_solr) [n:127.0.0.1:46551_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (11) -> (12)
[junit4] 2> 21053 INFO (zkCallback-64-thread-1-processing-n:127.0.0.1:50231_solr) [n:127.0.0.1:50231_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (11) -> (12)
[junit4] 2> 21053 INFO (zkCallback-57-thread-1-processing-n:127.0.0.1:50305_solr) [n:127.0.0.1:50305_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (11) -> (12)
[junit4] 2> 21053 INFO (zkCallback-58-thread-1-processing-n:127.0.0.1:55621_solr) [n:127.0.0.1:55621_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (11) -> (12)
[junit4] 2> 21053 INFO (zkCallback-77-thread-2-processing-n:127.0.0.1:59736_solr) [n:127.0.0.1:59736_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (11) -> (12)
[junit4] 2> 21053 INFO (zkCallback-66-thread-2-processing-n:127.0.0.1:49586_solr) [n:127.0.0.1:49586_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (11) -> (12)
[junit4] 2> 21053 INFO (zkCallback-78-thread-2-processing-n:127.0.0.1:51138_solr) [n:127.0.0.1:51138_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (11) -> (12)
[junit4] 2> 21054 INFO (zkCallback-72-thread-2-processing-n:127.0.0.1:57247_solr) [n:127.0.0.1:57247_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (11) -> (12)
[junit4] 2> 21417 INFO (jetty-launcher-17-thread-6) [n:127.0.0.1:49586_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node6/.
[junit4] 2> 21430 INFO (jetty-launcher-17-thread-4) [n:127.0.0.1:37157_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node4/.
[junit4] 2> 21439 INFO (jetty-launcher-17-thread-9) [n:127.0.0.1:48750_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node9/.
[junit4] 2> 21456 INFO (jetty-launcher-17-thread-5) [n:127.0.0.1:57247_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node5/.
[junit4] 2> 21462 INFO (jetty-launcher-17-thread-3) [n:127.0.0.1:33056_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node3/.
[junit4] 2> 21739 INFO (TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[9BBB7A8EADEC50B7]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (12)
[junit4] 2> 21749 INFO (TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[9BBB7A8EADEC50B7]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:47704/solr ready
[junit4] 2> 21825 INFO (qtp1906397125-212) [n:127.0.0.1:49586_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf1&maxShardsPerNode=4&name=collection1&action=CREATE&numShards=3&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 21868 INFO (OverseerThreadFactory-169-thread-1-processing-n:127.0.0.1:57247_solr) [n:127.0.0.1:57247_solr ] o.a.s.c.CreateCollectionCmd Create collection collection1
[junit4] 2> 22372 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=collection1_shard2_replica1&action=CREATE&numShards=3&collection=collection1&shard=shard2&wt=javabin&version=2
[junit4] 2> 22382 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=collection1_shard2_replica2&action=CREATE&numShards=3&collection=collection1&shard=shard2&wt=javabin&version=2
[junit4] 2> 22383 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=collection1_shard3_replica1&action=CREATE&numShards=3&collection=collection1&shard=shard3&wt=javabin&version=2
[junit4] 2> 22401 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=collection1_shard3_replica2&action=CREATE&numShards=3&collection=collection1&shard=shard3&wt=javabin&version=2
[junit4] 2> 22407 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=collection1_shard1_replica2&action=CREATE&numShards=3&collection=collection1&shard=shard1&wt=javabin&version=2
[junit4] 2> 22409 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=collection1_shard1_replica1&action=CREATE&numShards=3&collection=collection1&shard=shard1&wt=javabin&version=2
[junit4] 2> 22578 INFO (zkCallback-66-thread-2-processing-n:127.0.0.1:49586_solr) [n:127.0.0.1:49586_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [12])
[junit4] 2> 22578 INFO (zkCallback-74-thread-1-processing-n:127.0.0.1:37157_solr) [n:127.0.0.1:37157_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [12])
[junit4] 2> 22579 INFO (zkCallback-78-thread-2-processing-n:127.0.0.1:51138_solr) [n:127.0.0.1:51138_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [12])
[junit4] 2> 22579 INFO (zkCallback-75-thread-1-processing-n:127.0.0.1:46551_solr) [n:127.0.0.1:46551_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [12])
[junit4] 2> 22579 INFO (zkCallback-57-thread-1-processing-n:127.0.0.1:50305_solr) [n:127.0.0.1:50305_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [12])
[junit4] 2> 22579 INFO (zkCallback-76-thread-2-processing-n:127.0.0.1:33627_solr) [n:127.0.0.1:33627_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [12])
[junit4] 2> 23415 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
[junit4] 2> 23424 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
[junit4] 2> 23460 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
[junit4] 2> 23463 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.s.IndexSchema [collection1_shard2_replica2] Schema name=example
[junit4] 2> 23469 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
[junit4] 2> 23471 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
[junit4] 2> 23475 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.s.IndexSchema [collection1_shard3_replica1] Schema name=example
[junit4] 2> 23479 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
[junit4] 2> 23480 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.s.IndexSchema Loaded schema example/1.5 with uniqueid field id
[junit4] 2> 23507 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.s.IndexSchema [collection1_shard3_replica2] Schema name=example
[junit4] 2> 23508 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.s.IndexSchema Loaded schema example/1.5 with uniqueid field id
[junit4] 2> 23513 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard3_replica1' using configuration from collection collection1
[junit4] 2> 23513 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.c.SolrCore [[collection1_shard3_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node4/collection1_shard3_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node4/./collection1_shard3_replica1/data/]
[junit4] 2> 23516 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.s.IndexSchema Loaded schema example/1.5 with uniqueid field id
[junit4] 2> 23519 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.s.IndexSchema [collection1_shard2_replica1] Schema name=example
[junit4] 2> 23525 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.s.IndexSchema [collection1_shard1_replica1] Schema name=example
[junit4] 2> 23527 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica2' using configuration from collection collection1
[junit4] 2> 23527 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.c.SolrCore [[collection1_shard2_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node7/collection1_shard2_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node7/./collection1_shard2_replica2/data/]
[junit4] 2> 23532 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.s.IndexSchema [collection1_shard1_replica2] Schema name=example
[junit4] 2> 23543 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.s.IndexSchema Loaded schema example/1.5 with uniqueid field id
[junit4] 2> 23545 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard3_replica2' using configuration from collection collection1
[junit4] 2> 23547 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica1' using configuration from collection collection1
[junit4] 2> 23548 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.c.SolrCore [[collection1_shard2_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node2/collection1_shard2_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node2/./collection1_shard2_replica1/data/]
[junit4] 2> 23549 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.s.IndexSchema Loaded schema example/1.5 with uniqueid field id
[junit4] 2> 23556 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.c.SolrCore [[collection1_shard3_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node6/collection1_shard3_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node6/./collection1_shard3_replica2/data/]
[junit4] 2> 23579 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.s.IndexSchema Loaded schema example/1.5 with uniqueid field id
[junit4] 2> 23584 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica1' using configuration from collection collection1
[junit4] 2> 23584 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [[collection1_shard1_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node10/collection1_shard1_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node10/./collection1_shard1_replica1/data/]
[junit4] 2> 23601 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica2' using configuration from collection collection1
[junit4] 2> 23602 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.c.SolrCore [[collection1_shard1_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node1/collection1_shard1_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/contrib/solr-ltr/test/J2/temp/solr.ltr.TestLTROnSolrCloud_9BBB7A8EADEC50B7-001/tempDir-001/node1/./collection1_shard1_replica2/data/]
[junit4] 2> 23668 WARN (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.c.RequestHandlers no default request handler is registered (either '/select' or 'standard')
[junit4] 2> 23720 WARN (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.c.RequestHandlers no default request handler is registered (either '/select' or 'standard')
[junit4] 2> 23701 WARN (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.c.RequestHandlers no default request handler is registered (either '/select' or 'standard')
[junit4] 2> 23678 WARN (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.c.RequestHandlers no default request handler is registered (either '/select' or 'standard')
[junit4] 2> 23672 WARN (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.c.RequestHandlers no default request handler is registered (either '/select' or 'standard')
[junit4] 2> 23728 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 23729 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 23729 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 23729 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.u.CommitTracker Soft AutoCommit: if uncommited for 1000ms;
[junit4] 2> 23730 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@541b30fe[collection1_shard3_replica1] main]
[junit4] 2> 23733 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 23734 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23734 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.r.RestManager Registered ManagedResource impl org.apache.solr.ltr.store.rest.ManagedFeatureStore for path /schema/feature-store
[junit4] 2> 23734 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _schema_feature-store.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23734 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.l.s.r.ManagedFeatureStore ------ managed feature ~ loading ------
[junit4] 2> 23734 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.r.ManagedResource Notified 1 observers of /schema/feature-store
[junit4] 2> 23734 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.r.RestManager Registered new managed resource /schema/feature-store
[junit4] 2> 23734 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.r.RestManager Registered ManagedResource impl org.apache.solr.ltr.store.rest.ManagedModelStore for path /schema/model-store
[junit4] 2> 23735 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _schema_model-store.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23735 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.l.s.r.ManagedModelStore INIT model store
[junit4] 2> 23735 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.l.s.r.ManagedModelStore ------ managed models ~ loading ------
[junit4] 2> 23735 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.r.ManagedResource Notified 1 observers of /schema/model-store
[junit4] 2> 23735 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.r.RestManager Registered new managed resource /schema/model-store
[junit4] 2> 23735 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 23736 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 23736 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 23736 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559142526280007680
[junit4] 2> 23737 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 23737 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.u.CommitTracker Soft AutoCommit: if uncommited for 1000ms;
[junit4] 2> 23741 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@101856a5[collection1_shard2_replica2] main]
[junit4] 2> 23744 INFO (searcherExecutor-180-thread-1-processing-n:127.0.0.1:37157_solr x:collection1_shard3_replica1 s:shard3 c:collection1) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.c.SolrCore [collection1_shard3_replica1] Registered new searcher Searcher@541b30fe[collection1_shard3_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 23744 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 23745 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23745 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.r.RestManager Registered ManagedResource impl org.apache.solr.ltr.store.rest.ManagedFeatureStore for path /schema/feature-store
[junit4] 2> 23745 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _schema_feature-store.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23745 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.l.s.r.ManagedFeatureStore ------ managed feature ~ loading ------
[junit4] 2> 23745 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.r.ManagedResource Notified 1 observers of /schema/feature-store
[junit4] 2> 23745 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.r.RestManager Registered new managed resource /schema/feature-store
[junit4] 2> 23745 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.r.RestManager Registered ManagedResource impl org.apache.solr.ltr.store.rest.ManagedModelStore for path /schema/model-store
[junit4] 2> 23746 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _schema_model-store.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23746 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.l.s.r.ManagedModelStore INIT model store
[junit4] 2> 23746 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.l.s.r.ManagedModelStore ------ managed models ~ loading ------
[junit4] 2> 23746 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.r.ManagedResource Notified 1 observers of /schema/model-store
[junit4] 2> 23746 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.r.RestManager Registered new managed resource /schema/model-store
[junit4] 2> 23746 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 23746 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559142526290493440
[junit4] 2> 23756 INFO (searcherExecutor-181-thread-1-processing-n:127.0.0.1:33627_solr x:collection1_shard2_replica2 s:shard2 c:collection1) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.c.SolrCore [collection1_shard2_replica2] Registered new searcher Searcher@101856a5[collection1_shard2_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 23759 WARN (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.c.RequestHandlers no default request handler is registered (either '/select' or 'standard')
[junit4] 2> 23767 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 23767 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 23768 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 23768 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.u.CommitTracker Soft AutoCommit: if uncommited for 1000ms;
[junit4] 2> 23769 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@51431d58[collection1_shard3_replica2] main]
[junit4] 2> 23786 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 23786 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 23787 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 23787 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: if uncommited for 1000ms;
[junit4] 2> 23789 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 23789 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 23790 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 23790 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 23790 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 23790 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 23790 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: if uncommited for 1000ms;
[junit4] 2> 23791 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 23791 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.u.CommitTracker Soft AutoCommit: if uncommited for 1000ms;
[junit4] 2> 23792 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@24a3c6a[collection1_shard1_replica1] main]
[junit4] 2> 23792 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4842a116[collection1_shard2_replica1] main]
[junit4] 2> 23793 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23793 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@45ae427b[collection1_shard1_replica2] main]
[junit4] 2> 23793 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.r.RestManager Registered ManagedResource impl org.apache.solr.ltr.store.rest.ManagedFeatureStore for path /schema/feature-store
[junit4] 2> 23793 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _schema_feature-store.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23794 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.l.s.r.ManagedFeatureStore ------ managed feature ~ loading ------
[junit4] 2> 23794 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.r.ManagedResource Notified 1 observers of /schema/feature-store
[junit4] 2> 23794 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.r.RestManager Registered new managed resource /schema/feature-store
[junit4] 2> 23794 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.r.RestManager Registered ManagedResource impl org.apache.solr.ltr.store.rest.ManagedModelStore for path /schema/model-store
[junit4] 2> 23795 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _schema_model-store.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23795 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.l.s.r.ManagedModelStore INIT model store
[junit4] 2> 23795 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.l.s.r.ManagedModelStore ------ managed models ~ loading ------
[junit4] 2> 23795 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.r.ManagedResource Notified 1 observers of /schema/model-store
[junit4] 2> 23795 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.r.RestManager Registered new managed resource /schema/model-store
[junit4] 2> 23795 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 23795 INFO (qtp1906397125-214) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559142526341873664
[junit4] 2> 23796 INFO (searcherExecutor-182-thread-1-processing-n:127.0.0.1:49586_solr x:collection1_shard3_replica2 s:shard3 c:collection1) [n:127.0.0.1:49586_solr c:collection1 s:shard3 x:collection1_shard3_replica2] o.a.s.c.SolrCore [collection1_shard3_replica2] Registered new searcher Searcher@51431d58[collection1_shard3_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 23796 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 23797 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23797 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.r.RestManager Registered ManagedResource impl org.apache.solr.ltr.store.rest.ManagedFeatureStore for path /schema/feature-store
[junit4] 2> 23797 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 23798 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _schema_feature-store.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23798 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.l.s.r.ManagedFeatureStore ------ managed feature ~ loading ------
[junit4] 2> 23798 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.r.ManagedResource Notified 1 observers of /schema/feature-store
[junit4] 2> 23798 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.r.RestManager Registered new managed resource /schema/feature-store
[junit4] 2> 23798 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.r.RestManager Registered ManagedResource impl org.apache.solr.ltr.store.rest.ManagedModelStore for path /schema/model-store
[junit4] 2> 23798 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _schema_model-store.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23798 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.l.s.r.ManagedModelStore INIT model store
[junit4] 2> 23798 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.l.s.r.ManagedModelStore ------ managed models ~ loading ------
[junit4] 2> 23799 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.r.ManagedResource Notified 1 observers of /schema/model-store
[junit4] 2> 23799 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23799 INFO (qtp1000697022-231) [n:127.0.0.1:37157_solr c:collection1 s:shard3 x:collection1_shard3_replica1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard3: total=2 found=1 timeoutin=9999ms
[junit4] 2> 23799 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.r.RestManager Registered ManagedResource impl org.apache.solr.ltr.store.rest.ManagedFeatureStore for path /schema/feature-store
[junit4] 2> 23799 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.r.RestManager Registered new managed resource /schema/model-store
[junit4] 2> 23799 INFO (qtp1445269374-202) [n:127.0.0.1:33627_solr c:collection1 s:shard2 x:collection1_shard2_replica2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
[junit4] 2> 23799 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 23799 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 23799 INFO (qtp1743941054-265) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559142526346067968
[junit4] 2> 23799 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _schema_feature-store.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23799 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.l.s.r.ManagedFeatureStore ------ managed feature ~ loading ------
[junit4] 2> 23799 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.r.ManagedResource Notified 1 observers of /schema/feature-store
[junit4] 2> 23799 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.r.RestManager Registered new managed resource /schema/feature-store
[junit4] 2> 23799 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.r.RestManager Registered ManagedResource impl org.apache.solr.ltr.store.rest.ManagedModelStore for path /schema/model-store
[junit4] 2> 23799 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23799 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.r.RestManager Registered ManagedResource impl org.apache.solr.ltr.store.rest.ManagedFeatureStore for path /schema/feature-store
[junit4] 2> 23800 INFO (searcherExecutor-184-thread-1-processing-n:127.0.0.1:46551_solr x:collection1_shard1_replica1 s:shard1 c:collection1) [n:127.0.0.1:46551_solr c:collection1 s:shard1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1] Registered new searcher Searcher@24a3c6a[collection1_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 23802 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _schema_feature-store.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23802 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.l.s.r.ManagedFeatureStore ------ managed feature ~ loading ------
[junit4] 2> 23802 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.r.ManagedResource Notified 1 observers of /schema/feature-store
[junit4] 2> 23802 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.r.RestManager Registered new managed resource /schema/feature-store
[junit4] 2> 23802 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.r.RestManager Registered ManagedResource impl org.apache.solr.ltr.store.rest.ManagedModelStore for path /schema/model-store
[junit4] 2> 23802 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _schema_model-store.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23802 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.l.s.r.ManagedModelStore INIT model store
[junit4] 2> 23802 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.l.s.r.ManagedModelStore ------ managed models ~ loading ------
[junit4] 2> 23802 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.r.ManagedResource Notified 1 observers of /schema/model-store
[junit4] 2> 23802 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.r.RestManager Registered new managed resource /schema/model-store
[junit4] 2> 23802 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 23802 INFO (qtp1884834671-248) [n:127.0.0.1:50305_solr c:collection1 s:shard2 x:collection1_shard2_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559142526349213696
[junit4] 2> 23804 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _schema_model-store.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 23804 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.l.s.r.ManagedModelStore INIT model store
[junit4] 2> 23804 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.l.s.r.ManagedModelStore ------ managed models ~ loading ------
[junit4] 2> 23804 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.r.ManagedResource Notified 1 observers of /schema/model-store
[junit4] 2> 23804 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.r.RestManager Registered new managed resource /schema/model-store
[junit4] 2> 23804 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 23804 INFO (qtp1257172074-262) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collection1_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559142526351310848
[junit4] 2> 23805 INFO (searcherExecutor-185-thread-1-processing-n:127.0.0.1:51138_solr x:collection1_shard1_replica2 s:shard1 c:collection1) [n:127.0.0.1:51138_solr c:collection1 s:shard1 x:collecti
[...truncated too long message...]
org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:336)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:336)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:299)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.firstChild(DistributedQueue.java:279)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.firstElement(DistributedQueue.java:384)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:124)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:148)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 39852 INFO (OverseerStateUpdate-97446407584874527-127.0.0.1:37157_solr-n_0000000001) [n:127.0.0.1:37157_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37157_solr
[junit4] 2> 39875 WARN (zkCallback-78-thread-1-processing-n:127.0.0.1:51138_solr) [n:127.0.0.1:51138_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 39875 INFO (jetty-launcher-17-thread-6) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 39875 INFO (jetty-launcher-17-thread-6) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5f8fc541{/solr,null,UNAVAILABLE}
[junit4] 2> 39880 WARN (zkCallback-76-thread-3-processing-n:127.0.0.1:33627_solr) [n:127.0.0.1:33627_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 39880 INFO (jetty-launcher-17-thread-4) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 39880 INFO (jetty-launcher-17-thread-4) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@507a7af4{/solr,null,UNAVAILABLE}
[junit4] 2> 39881 WARN (zkCallback-75-thread-3-processing-n:127.0.0.1:46551_solr) [n:127.0.0.1:46551_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 39881 INFO (jetty-launcher-17-thread-12) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 39881 INFO (jetty-launcher-17-thread-12) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4f0e5ad5{/solr,null,UNAVAILABLE}
[junit4] 2> 39891 INFO (jetty-launcher-17-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 39891 INFO (jetty-launcher-17-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4e1a7b05{/solr,null,UNAVAILABLE}
[junit4] 2> 39895 INFO (TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[9BBB7A8EADEC50B7]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:47704 47704
[junit4] 2> 39994 INFO (Thread-45) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:47704 47704
[junit4] 2> 39996 WARN (Thread-45) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 14 /solr/aliases.json
[junit4] 2> 13 /solr/security.json
[junit4] 2> 7 /solr/configs/conf1
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 14 /solr/clusterprops.json
[junit4] 2> 14 /solr/clusterstate.json
[junit4] 2> 7 /solr/collections/collection1/state.json
[junit4] 2> 2 /solr/overseer_elect/election/97446407584874526-127.0.0.1:46551_solr-n_0000000008
[junit4] 2> 2 /solr/overseer_elect/election/97446407584874517-127.0.0.1:50305_solr-n_0000000004
[junit4] 2> 2 /solr/overseer_elect/election/97446407584874531-127.0.0.1:33627_solr-n_0000000003
[junit4] 2> 2 /solr/overseer_elect/election/97446407584874527-127.0.0.1:37157_solr-n_0000000001
[junit4] 2> 2 /solr/overseer_elect/election/97446407584874532-127.0.0.1:51138_solr-n_0000000009
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 14 /solr/live_nodes
[junit4] 2> 14 /solr/collections
[junit4] 2> 14 /solr/overseer/collection-queue-work
[junit4] 2> 13 /solr/overseer/queue
[junit4] 2> 5 /solr/overseer/queue-work
[junit4] 2>
[junit4] 2> 39996 INFO (TEST-TestLTROnSolrCloud.testSimpleQuery-seed#[9BBB7A8EADEC50B7]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testSimpleQuery
[junit4] 2> 40004 INFO (SUITE-TestLTROnSolrCloud-seed#[9BBB7A8EADEC50B7]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> 40004 INFO (SUITE-TestLTROnSolrCloud-seed#[9BBB7A8EADEC50B7]-worker) [ ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for all SolrIndexSearchers to be released
[junit4] 2> 40004 INFO (SUITE-TestLTROnSolrCloud-seed#[9BBB7A8EADEC50B7]-worker) [ ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
[junit4] 2> Feb 12, 2017 3:43:19 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 3 leaked thread(s).
[junit4] 2> 40349 WARN (OverseerCollectionConfigSetProcessor-97446407584874527-127.0.0.1:37157_solr-n_0000000001) [n:127.0.0.1:37157_solr ] o.a.s.c.OverseerTaskProcessor Overseer cannot talk to ZK
[junit4] 2> Feb 12, 2017 3:43:39 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.ltr.TestLTROnSolrCloud:
[junit4] 2> 1) Thread[id=601, name=OverseerHdfsCoreFailoverThread-97446407584874527-127.0.0.1:37157_solr-n_0000000001, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Feb 12, 2017 3:43:39 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: Starting to interrupt leaked threads:
[junit4] 2> 1) Thread[id=601, name=OverseerHdfsCoreFailoverThread-97446407584874527-127.0.0.1:37157_solr-n_0000000001, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
[junit4] 2> Feb 12, 2017 3:43:39 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: All leaked threads terminated.
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=false): {}, locale=is, timezone=Asia/Thimphu
[junit4] 2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=166872912,total=315621376
[junit4] 2> NOTE: All tests run in this JVM: [TestOriginalScoreScorer, TestRerankBase, TestLTRWithFacet, TestLinearModel, TestExternalFeatures, TestUserTermScorerQuery, TestLTROnSolrCloud]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestLTROnSolrCloud -Dtests.seed=9BBB7A8EADEC50B7 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=is -Dtests.timezone=Asia/Thimphu -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] ERROR 0.00s J2 | TestLTROnSolrCloud (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.ltr.TestLTROnSolrCloud:
[junit4] > 1) Thread[id=601, name=OverseerHdfsCoreFailoverThread-97446407584874527-127.0.0.1:37157_solr-n_0000000001, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > at __randomizedtesting.SeedInfo.seed([9BBB7A8EADEC50B7]:0)
[junit4] Completed [33/33 (1!)] on J2 in 41.12s, 1 test, 1 error <<< FAILURES!
[...truncated 55828 lines...]
-documentation-lint:
[jtidy] Checking for broken html (such as invalid tags)...
[delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/build/jtidy_tmp
[echo] Checking for broken links...
[exec]
[exec] Crawl/parse...
[exec]
[exec] Verify...
[echo] Checking for malformed docs...
[exec]
[exec] /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/docs/solr-solrj/overview-summary.html
[exec] missing: org.apache.solr.client.solrj.io.eval
[exec]
[exec] Missing javadocs were found!
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:775: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:101: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build.xml:654: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build.xml:670: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/common-build.xml:2493: exec returned: 1
Total time: 81 minutes 14 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-Tests-master - Build # 1659 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/1659/
1 tests failed.
FAILED: org.apache.solr.cloud.HttpPartitionTest.test
Error Message:
Captured an uncaught exception in thread: Thread[id=8130, name=SocketProxy-Response-46192:40805, state=RUNNABLE, group=TGRP-HttpPartitionTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=8130, name=SocketProxy-Response-46192:40805, state=RUNNABLE, group=TGRP-HttpPartitionTest]
at __randomizedtesting.SeedInfo.seed([41C1E877002A8B74:C995D7ADAED6E68C]:0)
Caused by: java.lang.RuntimeException: java.net.SocketException: Socket is closed
at __randomizedtesting.SeedInfo.seed([41C1E877002A8B74]:0)
at org.apache.solr.cloud.SocketProxy$Bridge$Pump.run(SocketProxy.java:347)
Caused by: java.net.SocketException: Socket is closed
at java.net.Socket.setSoTimeout(Socket.java:1137)
at org.apache.solr.cloud.SocketProxy$Bridge$Pump.run(SocketProxy.java:344)
Build Log:
[...truncated 11792 lines...]
[junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/init-core-data-001
[junit4] 2> 690787 INFO (SUITE-HttpPartitionTest-seed#[41C1E877002A8B74]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields
[junit4] 2> 690788 INFO (SUITE-HttpPartitionTest-seed#[41C1E877002A8B74]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
[junit4] 2> 690788 INFO (SUITE-HttpPartitionTest-seed#[41C1E877002A8B74]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /_/js
[junit4] 2> 690789 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 690792 INFO (Thread-2960) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 690792 INFO (Thread-2960) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 690892 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.ZkTestServer start zk server on port:50901
[junit4] 2> 690916 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 690918 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 690919 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 690919 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 690920 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 690921 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 690922 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 690922 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 690924 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 690925 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 690926 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 690997 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/control-001/cores/collection1
[junit4] 2> 691000 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 691000 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2392b4a0{/_/js,null,AVAILABLE}
[junit4] 2> 691001 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@3752cc7e{HTTP/1.1,[http/1.1]}{127.0.0.1:46192}
[junit4] 2> 691001 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.e.j.s.Server Started @694668ms
[junit4] 2> 691001 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/tempDir-001/control/data, hostContext=/_/js, hostPort=58800, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/control-001/cores}
[junit4] 2> 691001 ERROR (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 691001 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 691001 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 691001 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 691001 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-12T04:10:01.179Z
[junit4] 2> 691027 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 691027 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/control-001/solr.xml
[junit4] 2> 691056 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50901/solr
[junit4] 2> 691075 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [n:127.0.0.1:58800__%2Fjs ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:58800__%2Fjs
[junit4] 2> 691075 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [n:127.0.0.1:58800__%2Fjs ] o.a.s.c.Overseer Overseer (id=97443682777432068-127.0.0.1:58800__%2Fjs-n_0000000000) starting
[junit4] 2> 691097 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [n:127.0.0.1:58800__%2Fjs ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58800__%2Fjs
[junit4] 2> 691099 INFO (zkCallback-1074-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 691208 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [n:127.0.0.1:58800__%2Fjs ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/control-001/cores
[junit4] 2> 691208 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [n:127.0.0.1:58800__%2Fjs ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 691217 INFO (OverseerStateUpdate-97443682777432068-127.0.0.1:58800__%2Fjs-n_0000000000) [n:127.0.0.1:58800__%2Fjs ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 691237 INFO (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 691251 INFO (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 691332 WARN (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 691343 INFO (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 691351 INFO (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 691352 INFO (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/control-001/cores/collection1/data/]
[junit4] 2> 691352 INFO (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c0d8542
[junit4] 2> 691353 INFO (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=14, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 691380 WARN (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 691411 INFO (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 691411 INFO (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 691412 INFO (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 691412 INFO (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 691412 INFO (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=29, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.30749363195386503]
[junit4] 2> 691412 INFO (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@110f93d2[collection1] main]
[junit4] 2> 691413 INFO (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 691414 INFO (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 691414 INFO (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 691414 INFO (coreLoadExecutor-2745-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559098925086932992
[junit4] 2> 691417 INFO (searcherExecutor-2746-thread-1-processing-n:127.0.0.1:58800__%2Fjs x:collection1 c:control_collection) [n:127.0.0.1:58800__%2Fjs c:control_collection x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@110f93d2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 691423 INFO (coreZkRegister-2738-thread-1-processing-n:127.0.0.1:58800__%2Fjs x:collection1 c:control_collection) [n:127.0.0.1:58800__%2Fjs c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 691423 INFO (coreZkRegister-2738-thread-1-processing-n:127.0.0.1:58800__%2Fjs x:collection1 c:control_collection) [n:127.0.0.1:58800__%2Fjs c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 691423 INFO (coreZkRegister-2738-thread-1-processing-n:127.0.0.1:58800__%2Fjs x:collection1 c:control_collection) [n:127.0.0.1:58800__%2Fjs c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:58800/_/js/collection1/
[junit4] 2> 691423 INFO (coreZkRegister-2738-thread-1-processing-n:127.0.0.1:58800__%2Fjs x:collection1 c:control_collection) [n:127.0.0.1:58800__%2Fjs c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 691423 INFO (coreZkRegister-2738-thread-1-processing-n:127.0.0.1:58800__%2Fjs x:collection1 c:control_collection) [n:127.0.0.1:58800__%2Fjs c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:58800/_/js/collection1/ has no replicas
[junit4] 2> 691426 INFO (coreZkRegister-2738-thread-1-processing-n:127.0.0.1:58800__%2Fjs x:collection1 c:control_collection) [n:127.0.0.1:58800__%2Fjs c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:58800/_/js/collection1/ shard1
[junit4] 2> 691508 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 691509 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:50901/solr ready
[junit4] 2> 691509 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 691509 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
[junit4] 2> 691527 INFO (coreZkRegister-2738-thread-1-processing-n:127.0.0.1:58800__%2Fjs x:collection1 c:control_collection) [n:127.0.0.1:58800__%2Fjs c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 691588 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-1-001/cores/collection1
[junit4] 2> 691589 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-1-001
[junit4] 2> 691589 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 691590 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@72c9502b{/_/js,null,AVAILABLE}
[junit4] 2> 691591 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@354a189b{HTTP/1.1,[http/1.1]}{127.0.0.1:34073}
[junit4] 2> 691591 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.e.j.s.Server Started @695258ms
[junit4] 2> 691591 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/_/js, hostPort=40776, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-1-001/cores}
[junit4] 2> 691591 ERROR (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 691591 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 691591 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 691591 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 691591 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-12T04:10:01.769Z
[junit4] 2> 691594 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 691594 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-1-001/solr.xml
[junit4] 2> 691599 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50901/solr
[junit4] 2> 691606 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [n:127.0.0.1:40776__%2Fjs ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 691610 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [n:127.0.0.1:40776__%2Fjs ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40776__%2Fjs
[junit4] 2> 691611 INFO (zkCallback-1074-thread-3-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 691611 INFO (zkCallback-1078-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 691612 INFO (zkCallback-1084-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 691687 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [n:127.0.0.1:40776__%2Fjs ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-1-001/cores
[junit4] 2> 691687 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [n:127.0.0.1:40776__%2Fjs ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 691693 INFO (OverseerStateUpdate-97443682777432068-127.0.0.1:58800__%2Fjs-n_0000000000) [n:127.0.0.1:58800__%2Fjs ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
[junit4] 2> 691795 INFO (zkCallback-1084-thread-2-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 692709 INFO (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 692727 INFO (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 692810 WARN (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 692811 INFO (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 692818 INFO (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 692818 INFO (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-1-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-1-001/cores/collection1/data/]
[junit4] 2> 692818 INFO (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c0d8542
[junit4] 2> 692819 INFO (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=14, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 692831 WARN (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 692858 INFO (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 692858 INFO (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 692859 INFO (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 692859 INFO (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 692861 INFO (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=29, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.30749363195386503]
[junit4] 2> 692861 INFO (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1db9f742[collection1] main]
[junit4] 2> 692862 INFO (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 692862 INFO (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 692862 INFO (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 692864 INFO (searcherExecutor-2757-thread-1-processing-n:127.0.0.1:40776__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1db9f742[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 692864 INFO (coreLoadExecutor-2756-thread-1-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559098926607368192
[junit4] 2> 692869 INFO (coreZkRegister-2751-thread-1-processing-n:127.0.0.1:40776__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:40776__%2Fjs c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 692869 INFO (coreZkRegister-2751-thread-1-processing-n:127.0.0.1:40776__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:40776__%2Fjs c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 692869 INFO (coreZkRegister-2751-thread-1-processing-n:127.0.0.1:40776__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:40776__%2Fjs c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40776/_/js/collection1/
[junit4] 2> 692869 INFO (coreZkRegister-2751-thread-1-processing-n:127.0.0.1:40776__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:40776__%2Fjs c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 692869 INFO (coreZkRegister-2751-thread-1-processing-n:127.0.0.1:40776__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:40776__%2Fjs c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:40776/_/js/collection1/ has no replicas
[junit4] 2> 692873 INFO (coreZkRegister-2751-thread-1-processing-n:127.0.0.1:40776__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:40776__%2Fjs c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40776/_/js/collection1/ shard2
[junit4] 2> 692976 INFO (zkCallback-1084-thread-2-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 693024 INFO (coreZkRegister-2751-thread-1-processing-n:127.0.0.1:40776__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:40776__%2Fjs c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 693105 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-2-001/cores/collection1
[junit4] 2> 693106 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-2-001
[junit4] 2> 693106 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 693107 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@28b15b7b{/_/js,null,AVAILABLE}
[junit4] 2> 693107 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@3d7fc84a{HTTP/1.1,[http/1.1]}{127.0.0.1:53765}
[junit4] 2> 693107 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.e.j.s.Server Started @696775ms
[junit4] 2> 693107 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/_/js, hostPort=46974, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-2-001/cores}
[junit4] 2> 693108 ERROR (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 693108 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 693108 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 693108 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 693108 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-12T04:10:03.286Z
[junit4] 2> 693111 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 693111 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-2-001/solr.xml
[junit4] 2> 693116 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50901/solr
[junit4] 2> 693124 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [n:127.0.0.1:46974__%2Fjs ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 693127 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [n:127.0.0.1:46974__%2Fjs ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46974__%2Fjs
[junit4] 2> 693128 INFO (zkCallback-1084-thread-2-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 693128 INFO (zkCallback-1078-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 693129 INFO (zkCallback-1074-thread-3-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 693129 INFO (zkCallback-1084-thread-2-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 693136 INFO (zkCallback-1090-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 693175 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [n:127.0.0.1:46974__%2Fjs ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-2-001/cores
[junit4] 2> 693175 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [n:127.0.0.1:46974__%2Fjs ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 693181 INFO (OverseerStateUpdate-97443682777432068-127.0.0.1:58800__%2Fjs-n_0000000000) [n:127.0.0.1:58800__%2Fjs ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 693282 WARN (OverseerStateUpdate-97443682777432068-127.0.0.1:58800__%2Fjs-n_0000000000) [n:127.0.0.1:58800__%2Fjs ] o.a.s.c.Overseer Bad version writing to ZK using compare-and-set, will force refresh cluster state: KeeperErrorCode = BadVersion for /collections/collection1/state.json
[junit4] 2> 693285 INFO (zkCallback-1090-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 693285 INFO (zkCallback-1084-thread-2-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 693286 INFO (OverseerStateUpdate-97443682777432068-127.0.0.1:58800__%2Fjs-n_0000000000) [n:127.0.0.1:58800__%2Fjs ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 693286 INFO (zkCallback-1090-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 693286 INFO (zkCallback-1084-thread-2-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 694190 INFO (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 694201 INFO (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 694283 WARN (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 694285 INFO (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 694292 INFO (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 694293 INFO (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-2-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-2-001/cores/collection1/data/]
[junit4] 2> 694293 INFO (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c0d8542
[junit4] 2> 694294 INFO (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=14, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 694298 WARN (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 694324 INFO (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 694324 INFO (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 694325 INFO (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 694325 INFO (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 694325 INFO (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=29, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.30749363195386503]
[junit4] 2> 694326 INFO (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7bd20a21[collection1] main]
[junit4] 2> 694326 INFO (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 694327 INFO (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 694327 INFO (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 694327 INFO (coreLoadExecutor-2767-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559098928141434880
[junit4] 2> 694332 INFO (searcherExecutor-2768-thread-1-processing-n:127.0.0.1:46974__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:46974__%2Fjs c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7bd20a21[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 694333 INFO (coreZkRegister-2762-thread-1-processing-n:127.0.0.1:46974__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:46974__%2Fjs c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 694333 INFO (coreZkRegister-2762-thread-1-processing-n:127.0.0.1:46974__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:46974__%2Fjs c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 694333 INFO (coreZkRegister-2762-thread-1-processing-n:127.0.0.1:46974__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:46974__%2Fjs c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46974/_/js/collection1/
[junit4] 2> 694333 INFO (coreZkRegister-2762-thread-1-processing-n:127.0.0.1:46974__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:46974__%2Fjs c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 694333 INFO (coreZkRegister-2762-thread-1-processing-n:127.0.0.1:46974__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:46974__%2Fjs c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:46974/_/js/collection1/ has no replicas
[junit4] 2> 694336 INFO (coreZkRegister-2762-thread-1-processing-n:127.0.0.1:46974__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:46974__%2Fjs c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46974/_/js/collection1/ shard1
[junit4] 2> 694438 INFO (zkCallback-1084-thread-2-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 694438 INFO (zkCallback-1090-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 694461 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-3-001/cores/collection1
[junit4] 2> 694462 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-3-001
[junit4] 2> 694462 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 694463 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@162640d2{/_/js,null,AVAILABLE}
[junit4] 2> 694463 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@25d6ee56{HTTP/1.1,[http/1.1]}{127.0.0.1:34217}
[junit4] 2> 694463 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.e.j.s.Server Started @698131ms
[junit4] 2> 694463 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/_/js, hostPort=38608, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-3-001/cores}
[junit4] 2> 694463 ERROR (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 694464 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 694464 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 694464 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 694464 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-12T04:10:04.642Z
[junit4] 2> 694466 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 694466 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-3-001/solr.xml
[junit4] 2> 694471 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50901/solr
[junit4] 2> 694477 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [n:127.0.0.1:38608__%2Fjs ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 694479 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [n:127.0.0.1:38608__%2Fjs ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38608__%2Fjs
[junit4] 2> 694480 INFO (zkCallback-1084-thread-2-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 694481 INFO (zkCallback-1090-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 694480 INFO (zkCallback-1074-thread-2-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 694481 INFO (zkCallback-1078-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 694489 INFO (coreZkRegister-2762-thread-1-processing-n:127.0.0.1:46974__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:46974__%2Fjs c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 694492 INFO (zkCallback-1096-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 694530 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [n:127.0.0.1:38608__%2Fjs ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-3-001/cores
[junit4] 2> 694530 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [n:127.0.0.1:38608__%2Fjs ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 694532 INFO (OverseerStateUpdate-97443682777432068-127.0.0.1:58800__%2Fjs-n_0000000000) [n:127.0.0.1:58800__%2Fjs ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
[junit4] 2> 694634 INFO (zkCallback-1084-thread-2-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 694634 INFO (zkCallback-1096-thread-2-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 694634 INFO (zkCallback-1090-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 695552 INFO (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 695564 INFO (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 695642 WARN (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 695644 INFO (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 695654 INFO (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 695655 INFO (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-3-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-3-001/cores/collection1/data/]
[junit4] 2> 695655 INFO (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c0d8542
[junit4] 2> 695656 INFO (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=14, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 695665 WARN (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 695711 INFO (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 695711 INFO (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 695712 INFO (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 695712 INFO (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 695713 INFO (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=29, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.30749363195386503]
[junit4] 2> 695713 INFO (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@66650914[collection1] main]
[junit4] 2> 695714 INFO (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 695714 INFO (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 695715 INFO (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 695716 INFO (coreLoadExecutor-2778-thread-1-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559098929597906944
[junit4] 2> 695718 INFO (searcherExecutor-2779-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:38608__%2Fjs c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@66650914[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 695721 INFO (coreZkRegister-2773-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 c:collection1) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
[junit4] 2> 695724 INFO (updateExecutor-1093-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 695725 INFO (recoveryExecutor-1094-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 695725 INFO (recoveryExecutor-1094-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 695725 INFO (recoveryExecutor-1094-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
[junit4] 2> 695725 INFO (recoveryExecutor-1094-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 695725 INFO (recoveryExecutor-1094-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:40776/_/js/collection1/] and I am [http://127.0.0.1:38608/_/js/collection1/]
[junit4] 2> 695733 INFO (recoveryExecutor-1094-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:40776/_/js]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:38608__%252Fjs&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 695733 INFO (SocketProxy-Acceptor-40776) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=33987,localport=40776], receiveBufferSize:531000
[junit4] 2> 695736 INFO (SocketProxy-Acceptor-40776) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=34073,localport=43935], receiveBufferSize=530904
[junit4] 2> 695737 INFO (qtp1065567383-7649) [n:127.0.0.1:40776__%2Fjs ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 695737 INFO (qtp1065567383-7649) [n:127.0.0.1:40776__%2Fjs ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard2 of collection1) have state: recovering
[junit4] 2> 695737 INFO (qtp1065567383-7649) [n:127.0.0.1:40776__%2Fjs ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard2, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:38608__%2Fjs, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:38608/_/js","node_name":"127.0.0.1:38608__%2Fjs","state":"down"}
[junit4] 2> 695770 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.SolrTestCaseJ4 ###Starting test
[junit4] 2> 695770 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Wait for recoveries to finish - wait 30000 for each attempt
[junit4] 2> 695770 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30000
[junit4] 2> 695828 INFO (zkCallback-1084-thread-2-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 695828 INFO (zkCallback-1096-thread-2-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 695828 INFO (zkCallback-1090-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 696738 INFO (qtp1065567383-7649) [n:127.0.0.1:40776__%2Fjs ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard2, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:38608__%2Fjs, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:38608/_/js","node_name":"127.0.0.1:38608__%2Fjs","state":"recovering"}
[junit4] 2> 696738 INFO (qtp1065567383-7649) [n:127.0.0.1:40776__%2Fjs ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 696738 INFO (qtp1065567383-7649) [n:127.0.0.1:40776__%2Fjs ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:38608__%252Fjs&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node3&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1001
[junit4] 2> 703739 INFO (recoveryExecutor-1094-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:40776/_/js/collection1/] - recoveringAfterStartup=[true]
[junit4] 2> 703739 INFO (recoveryExecutor-1094-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=http://127.0.0.1:38608/_/js START replicas=[http://127.0.0.1:40776/_/js/collection1/] nUpdates=100
[junit4] 2> 703740 INFO (SocketProxy-Acceptor-40776) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=34064,localport=40776], receiveBufferSize:531000
[junit4] 2> 703740 INFO (SocketProxy-Acceptor-40776) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=34073,localport=44012], receiveBufferSize=530904
[junit4] 2> 703748 INFO (qtp1065567383-7655) [n:127.0.0.1:40776__%2Fjs c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 703748 INFO (qtp1065567383-7655) [n:127.0.0.1:40776__%2Fjs c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1] webapp=/_/js path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 703749 INFO (recoveryExecutor-1094-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 703749 INFO (recoveryExecutor-1094-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
[junit4] 2> 703749 INFO (recoveryExecutor-1094-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 703749 INFO (recoveryExecutor-1094-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 703750 INFO (recoveryExecutor-1094-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 703750 INFO (recoveryExecutor-1094-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
[junit4] 2> 703750 INFO (recoveryExecutor-1094-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
[junit4] 2> 703750 INFO (recoveryExecutor-1094-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
[junit4] 2> 703750 INFO (recoveryExecutor-1094-thread-1-processing-n:127.0.0.1:38608__%2Fjs x:collection1 s:shard2 c:collection1 r:core_node3) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
[junit4] 2> 703751 INFO (zkCallback-1084-thread-2-processing-n:127.0.0.1:40776__%2Fjs) [n:127.0.0.1:40776__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 703752 INFO (zkCallback-1090-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 703752 INFO (zkCallback-1096-thread-2-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 703774 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
[junit4] 2> 703774 INFO (SocketProxy-Acceptor-58800) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=40244,localport=58800], receiveBufferSize:531000
[junit4] 2> 703775 INFO (SocketProxy-Acceptor-58800) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=46192,localport=54887], receiveBufferSize=530904
[junit4] 2> 703777 INFO (qtp1228904418-7609) [n:127.0.0.1:58800__%2Fjs c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 703780 INFO (qtp1228904418-7609) [n:127.0.0.1:58800__%2Fjs c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 703781 INFO (qtp1228904418-7609) [n:127.0.0.1:58800__%2Fjs c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 703781 INFO (qtp1228904418-7609) [n:127.0.0.1:58800__%2Fjs c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_/js path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 3
[junit4] 2> 703782 INFO (SocketProxy-Acceptor-40776) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=34068,localport=40776], receiveBufferSize:531000
[junit4] 2> 703782 INFO (SocketProxy-Acceptor-40776) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=34073,localport=44016], receiveBufferSize=530904
[junit4] 2> 703790 INFO (SocketProxy-Acceptor-46974) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=47296,localport=46974], receiveBufferSize:531000
[junit4] 2> 703790 INFO (SocketProxy-Acceptor-40776) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=34072,localport=40776], receiveBufferSize:531000
[junit4] 2> 703791 INFO (SocketProxy-Acceptor-38608) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=53069,localport=38608], receiveBufferSize:531000
[junit4] 2> 703792 INFO (SocketProxy-Acceptor-40776) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=34073,localport=44020], receiveBufferSize=530904
[junit4] 2> 703792 INFO (SocketProxy-Acceptor-46974) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=53765,localport=48994], receiveBufferSize=530904
[junit4] 2> 703800 INFO (SocketProxy-Acceptor-38608) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=34217,localport=59560], receiveBufferSize=530904
[junit4] 2> 703800 INFO (qtp1065567383-7648) [n:127.0.0.1:40776__%2Fjs c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 703800 INFO (qtp1065567383-7648) [n:127.0.0.1:40776__%2Fjs c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 703803 INFO (qtp1831672037-7681) [n:127.0.0.1:46974__%2Fjs c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 703806 INFO (qtp1109339372-7711) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 703806 INFO (qtp1109339372-7711) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 703806 INFO (qtp1109339372-7711) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 703806 INFO (qtp1109339372-7711) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_/js path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:40776/_/js/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 703809 INFO (qtp1831672037-7681) [n:127.0.0.1:46974__%2Fjs c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 703809 INFO (qtp1831672037-7681) [n:127.0.0.1:46974__%2Fjs c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 703809 INFO (qtp1065567383-7648) [n:127.0.0.1:40776__%2Fjs c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 703810 INFO (qtp1065567383-7648) [n:127.0.0.1:40776__%2Fjs c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_/js path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:40776/_/js/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 9
[junit4] 2> 703810 INFO (qtp1831672037-7681) [n:127.0.0.1:46974__%2Fjs c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_/js path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:40776/_/js/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 6
[junit4] 2> 703811 INFO (qtp1065567383-7649) [n:127.0.0.1:40776__%2Fjs c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_/js path=/update params={_stateVer_=collection1:9&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 22
[junit4] 2> 703813 INFO (SocketProxy-Acceptor-40776) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=34076,localport=40776], receiveBufferSize:531000
[junit4] 2> 703816 INFO (qtp1065567383-7652) [n:127.0.0.1:40776__%2Fjs c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1] webapp=/_/js path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 703816 INFO (SocketProxy-Acceptor-40776) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=34073,localport=44024], receiveBufferSize=530904
[junit4] 2> 703821 INFO (SocketProxy-Acceptor-38608) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=53073,localport=38608], receiveBufferSize:531000
[junit4] 2> 703822 INFO (SocketProxy-Acceptor-38608) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=34217,localport=59564], receiveBufferSize=530904
[junit4] 2> 703828 INFO (qtp1109339372-7713) [n:127.0.0.1:38608__%2Fjs c:collection1 s:shard2 r:core_node3 x:collection1] o.a.s.c.S.Request [collection1] webapp=/_/js path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 703829 INFO (SocketProxy-Acceptor-46974) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=47306,localport=46974], receiveBufferSize:531000
[junit4] 2> 703830 INFO (SocketProxy-Acceptor-46974) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=53765,localport=49004], receiveBufferSize=530904
[junit4] 2> 703836 INFO (qtp1831672037-7683) [n:127.0.0.1:46974__%2Fjs c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.S.Request [collection1] webapp=/_/js path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 705838 INFO (SocketProxy-Acceptor-40776) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=34103,localport=40776], receiveBufferSize:531000
[junit4] 2> 705838 INFO (SocketProxy-Acceptor-40776) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=34073,localport=44051], receiveBufferSize=530904
[junit4] 2> 705839 INFO (qtp1065567383-7650) [n:127.0.0.1:40776__%2Fjs ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&maxShardsPerNode=1&name=c8n_crud_1x2&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 705841 INFO (OverseerThreadFactory-2743-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs ] o.a.s.c.CreateCollectionCmd Create collection c8n_crud_1x2
[junit4] 2> 705842 INFO (OverseerThreadFactory-2743-thread-1-processing-n:127.0.0.1:58800__%2Fjs) [n:127.0.0.1:58800__%2Fjs ] o.a.s.c.CreateCollectionCmd Only one config set found in zk - using it:conf1
[junit4] 2> 705945 INFO (SocketProxy-Acceptor-38608) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=53101,localport=38608], receiveBufferSize:531000
[junit4] 2> 705945 INFO (SocketProxy-Acceptor-46974) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=47331,localport=46974], receiveBufferSize:531000
[junit4] 2> 705945 INFO (SocketProxy-Acceptor-46974) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=53765,localport=49031], receiveBufferSize=530904
[junit4] 2> 705947 INFO (qtp1831672037-7684) [n:127.0.0.1:46974__%2Fjs ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=c8n_crud_1x2_shard1_replica2&action=CREATE&numShards=1&collection=c8n_crud_1x2&shard=shard1&wt=javabin&version=2
[junit4] 2> 705947 INFO (SocketProxy-Acceptor-38608) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=34217,localport=59592], receiveBufferSize=530904
[junit4] 2> 705948 INFO (qtp1109339372-7713) [n:127.0.0.1:38608__%2Fjs ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=c8n_crud_1x2_shard1_replica1&action=CREATE&numShards=1&collection=c8n_crud_1x2&shard=shard1&wt=javabin&version=2
[junit4] 2> 706052 INFO (zkCallback-1090-thread-1-processing-n:127.0.0.1:46974__%2Fjs) [n:127.0.0.1:46974__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/c8n_crud_1x2/state.json] for collection [c8n_crud_1x2] has occurred - updating... (live nodes size: [4])
[junit4] 2> 706052 INFO (zkCallback-1096-thread-2-processing-n:127.0.0.1:38608__%2Fjs) [n:127.0.0.1:38608__%2Fjs ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/c8n_crud_1x2/state.json] for collection [c8n_crud_1x2] has occurred - updating... (live nodes size: [4])
[junit4] 2> 706959 INFO (qtp1831672037-7684) [n:127.0.0.1:46974__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 706962 INFO (qtp1109339372-7713) [n:127.0.0.1:38608__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 706970 INFO (qtp1831672037-7684) [n:127.0.0.1:46974__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica2] o.a.s.s.IndexSchema [c8n_crud_1x2_shard1_replica2] Schema name=test
[junit4] 2> 706972 INFO (qtp1109339372-7713) [n:127.0.0.1:38608__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica1] o.a.s.s.IndexSchema [c8n_crud_1x2_shard1_replica1] Schema name=test
[junit4] 2> 707050 WARN (qtp1831672037-7684) [n:127.0.0.1:46974__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica2] o.a.s.s.IndexSchema [c8n_crud_1x2_shard1_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 707052 INFO (qtp1831672037-7684) [n:127.0.0.1:46974__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica2] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 707057 WARN (qtp1109339372-7713) [n:127.0.0.1:38608__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica1] o.a.s.s.IndexSchema [c8n_crud_1x2_shard1_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 707059 INFO (qtp1109339372-7713) [n:127.0.0.1:38608__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 707060 INFO (qtp1831672037-7684) [n:127.0.0.1:46974__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'c8n_crud_1x2_shard1_replica2' using configuration from collection c8n_crud_1x2
[junit4] 2> 707061 INFO (qtp1831672037-7684) [n:127.0.0.1:46974__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica2] o.a.s.c.SolrCore [[c8n_crud_1x2_shard1_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-2-001/cores/c8n_crud_1x2_shard1_replica2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-2-001/cores/c8n_crud_1x2_shard1_replica2/data/]
[junit4] 2> 707061 INFO (qtp1831672037-7684) [n:127.0.0.1:46974__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica2] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c0d8542
[junit4] 2> 707063 INFO (qtp1831672037-7684) [n:127.0.0.1:46974__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=14, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 707067 INFO (qtp1109339372-7713) [n:127.0.0.1:38608__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'c8n_crud_1x2_shard1_replica1' using configuration from collection c8n_crud_1x2
[junit4] 2> 707067 INFO (qtp1109339372-7713) [n:127.0.0.1:38608__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica1] o.a.s.c.SolrCore [[c8n_crud_1x2_shard1_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-3-001/cores/c8n_crud_1x2_shard1_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest_41C1E877002A8B74-001/shard-3-001/cores/c8n_crud_1x2_shard1_replica1/data/]
[junit4] 2> 707067 INFO (qtp1109339372-7713) [n:127.0.0.1:38608__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c0d8542
[junit4] 2> 707068 INFO (qtp1109339372-7713) [n:127.0.0.1:38608__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=14, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 707076 WARN (qtp1109339372-7713) [n:127.0.0.1:38608__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 707080 WARN (qtp1831672037-7684) [n:127.0.0.1:46974__%2Fjs c:c8n_crud_1x2 s:shard1 x:c8n_crud_1x2_shard1_replica2] o.a.s.c.RequestHandlers INVALID paramSet a in r
[...truncated too long message...]
ontextHandler Stopped o.e.j.s.ServletContextHandler@162640d2{/_/js,null,UNAVAILABLE}
[junit4] 2> 809828 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.ChaosMonkey monkey: stop shard! 58800
[junit4] 2> 809829 INFO (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:50901 50901
[junit4] 2> 809916 INFO (Thread-2960) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:50901 50901
[junit4] 2> 809916 WARN (Thread-2960) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/clusterprops.json
[junit4] 2> 6 /solr/aliases.json
[junit4] 2> 5 /solr/security.json
[junit4] 2> 5 /solr/configs/conf1
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/clusterstate.json
[junit4] 2> 4 /solr/collections/collection1/state.json
[junit4] 2> 4 /solr/collections/collMinRf_1x3/state.json
[junit4] 2> 3 /solr/collections/c8n_1x3/state.json
[junit4] 2> 3 /solr/collections/c8n_1x2_leader_session_loss/state.json
[junit4] 2> 2 /solr/collections/c8n_1x2/state.json
[junit4] 2> 2 /solr/collections/c8n_crud_1x2/state.json
[junit4] 2> 2 /solr/overseer_elect/election/97443682777432073-127.0.0.1:40776__%2Fjs-n_0000000001
[junit4] 2> 2 /solr/collections/collMinRf_1x3/leader_elect/shard1/election/97443682777432079-core_node1-n_0000000001
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 100 /solr/overseer/collection-queue-work
[junit4] 2> 97 /solr/overseer/queue
[junit4] 2> 33 /solr/overseer/queue-work
[junit4] 2> 7 /solr/live_nodes
[junit4] 2> 6 /solr/collections
[junit4] 2>
[junit4] 2> 809916 WARN (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.SocketProxy Closing 17 connections to: http://127.0.0.1:58800/_/js, target: http://127.0.0.1:46192/_/js
[junit4] 2> 809916 WARN (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.SocketProxy Closing 14 connections to: http://127.0.0.1:40776/_/js, target: http://127.0.0.1:34073/_/js
[junit4] 2> 809916 WARN (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.SocketProxy Closing 8 connections to: http://127.0.0.1:46974/_/js, target: http://127.0.0.1:53765/_/js
[junit4] 2> 809917 WARN (TEST-HttpPartitionTest.test-seed#[41C1E877002A8B74]) [ ] o.a.s.c.SocketProxy Closing 7 connections to: http://127.0.0.1:38608/_/js, target: http://127.0.0.1:34217/_/js
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=HttpPartitionTest -Dtests.method=test -Dtests.seed=41C1E877002A8B74 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=de -Dtests.timezone=America/Marigot -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 119s J1 | HttpPartitionTest.test <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=8130, name=SocketProxy-Response-46192:40805, state=RUNNABLE, group=TGRP-HttpPartitionTest]
[junit4] > at __randomizedtesting.SeedInfo.seed([41C1E877002A8B74:C995D7ADAED6E68C]:0)
[junit4] > Caused by: java.lang.RuntimeException: java.net.SocketException: Socket is closed
[junit4] > at __randomizedtesting.SeedInfo.seed([41C1E877002A8B74]:0)
[junit4] > at org.apache.solr.cloud.SocketProxy$Bridge$Pump.run(SocketProxy.java:347)
[junit4] > Caused by: java.net.SocketException: Socket is closed
[junit4] > at java.net.Socket.setSoTimeout(Socket.java:1137)
[junit4] > at org.apache.solr.cloud.SocketProxy$Bridge$Pump.run(SocketProxy.java:344)
[junit4] 2> 809919 INFO (SUITE-HttpPartitionTest-seed#[41C1E877002A8B74]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> 809919 INFO (SUITE-HttpPartitionTest-seed#[41C1E877002A8B74]-worker) [ ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for all SolrIndexSearchers to be released
[junit4] 2> 809919 INFO (SUITE-HttpPartitionTest-seed#[41C1E877002A8B74]-worker) [ ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
[junit4] 2> Feb 12, 2017 4:12:00 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {range_facet_l_dv=BlockTreeOrds(blocksize=128), multiDefault=PostingsFormat(name=Memory), a_t=Lucene50(blocksize=128), intDefault=Lucene50(blocksize=128), id=BlockTreeOrds(blocksize=128), range_facet_i_dv=Lucene50(blocksize=128), text=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), range_facet_l=Lucene50(blocksize=128), timestamp=Lucene50(blocksize=128)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Asserting), _version_=DocValuesFormat(name=Lucene70), range_facet_i_dv=DocValuesFormat(name=Lucene70), intDvoDefault=DocValuesFormat(name=Direct), timestamp=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=333, maxMBSortInHeap=7.709536367470244, sim=RandomSimilarity(queryNorm=true): {}, locale=de, timezone=America/Marigot
[junit4] 2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=4,threads=1,free=271209720,total=508559360
[junit4] 2> NOTE: All tests run in this JVM: [TestScoreJoinQPScore, TestSolrCLIRunExample, DirectSolrConnectionTest, TestReloadAndDeleteDocs, TestGraphMLResponseWriter, TestChildDocTransformer, SampleTest, TestStressUserVersions, IgnoreCommitOptimizeUpdateProcessorFactoryTest, AsyncCallRequestStatusResponseTest, UpdateLogTest, DocValuesTest, BigEndianAscendingWordDeserializerTest, CloudExitableDirectoryReaderTest, ManagedSchemaRoundRobinCloudTest, TestRandomCollapseQParserPlugin, TestEmbeddedSolrServerConstructors, BlockJoinFacetSimpleTest, TestSchemaManager, TestSurroundQueryParser, HLLUtilTest, ExitableDirectoryReaderTest, HttpSolrCallGetCoreTest, StressHdfsTest, ClassificationUpdateProcessorTest, TestCopyFieldCollectionResource, TestConfigReload, OverseerModifyCollectionTest, TestManagedSchemaAPI, SuggestComponentContextFilterQueryTest, LoggingHandlerTest, MetricsHandlerTest, TestDocumentBuilder, StatsReloadRaceTest, TestUninvertingReader, TestIntervalFaceting, VMParamsZkACLAndCredentialsProvidersTest, BasicAuthStandaloneTest, TestElisionMultitermQuery, DistanceFunctionTest, SolrGangliaReporterTest, DistributedMLTComponentTest, SOLR749Test, TestPartialUpdateDeduplication, HighlighterTest, RulesTest, ExplicitHLLTest, QueryParsingTest, AlternateDirectoryTest, TestSolrCloudWithKerberosAlt, TestShortCircuitedRequests, TestFreeTextSuggestions, BasicFunctionalityTest, TestCursorMarkWithoutUniqueKey, DistributedIntervalFacetingTest, DistributedVersionInfoTest, TestXmlQParserPlugin, TestLegacyNumericRangeQueryBuilder, TestSolrDynamicMBean, SharedFSAutoReplicaFailoverUtilsTest, CreateCollectionCleanupTest, TestSolrCoreParser, ConcurrentDeleteAndCreateCollectionTest, FileUtilsTest, SolrSlf4jReporterTest, TermVectorComponentTest, DistributedExpandComponentTest, TestNonDefinedSimilarityFactory, WordBreakSolrSpellCheckerTest, TestCloudPseudoReturnFields, TestLeaderInitiatedRecoveryThread, TestBulkSchemaConcurrent, HdfsChaosMonkeyNothingIsSafeTest, SaslZkACLProviderTest, TestExactStatsCache, TestLegacyFieldCache, TestCustomSort, TestDelegationWithHadoopAuth, CdcrReplicationDistributedZkTest, ChaosMonkeySafeLeaderTest, TestXIncludeConfig, CollectionTooManyReplicasTest, TestMinMaxOnMultiValuedField, BlockDirectoryTest, TestSolrIndexConfig, TestSystemCollAutoCreate, TolerantUpdateProcessorTest, TestUseDocValuesAsStored2, TestFieldCacheReopen, ReplicaListTransformerTest, OverseerTaskQueueTest, CSVRequestHandlerTest, DOMUtilTest, IndexBasedSpellCheckerTest, TestRawResponseWriter, TestFuzzyAnalyzedSuggestions, TestDistribDocBasedVersion, SuggesterFSTTest, SimplePostToolTest, TestSQLHandlerNonCloud, DeleteLastCustomShardedReplicaTest, SolrCmdDistributorTest, TestCloudDeleteByQuery, SolrInfoMBeanTest, TimeZoneUtilsTest, HttpPartitionTest]
[junit4] Completed [287/694 (1!)] on J1 in 120.25s, 1 test, 1 error <<< FAILURES!
[...truncated 63462 lines...]
-documentation-lint:
[jtidy] Checking for broken html (such as invalid tags)...
[delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/build/jtidy_tmp
[echo] Checking for broken links...
[exec]
[exec] Crawl/parse...
[exec]
[exec] Verify...
[echo] Checking for malformed docs...
[exec]
[exec] /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/docs/solr-solrj/overview-summary.html
[exec] missing: org.apache.solr.client.solrj.io.eval
[exec]
[exec] Missing javadocs were found!
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:775: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:101: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build.xml:654: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build.xml:670: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/common-build.xml:2493: exec returned: 1
Total time: 80 minutes 43 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-Tests-master - Build # 1658 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/1658/
1 tests failed.
FAILED: org.apache.solr.cloud.PeerSyncReplicationTest.test
Error Message:
timeout waiting to see all nodes active
Stack Trace:
java.lang.AssertionError: timeout waiting to see all nodes active
at __randomizedtesting.SeedInfo.seed([FAB2F37EA14E8E10:72E6CCA40FB2E3E8]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.PeerSyncReplicationTest.waitTillNodesActive(PeerSyncReplicationTest.java:326)
at org.apache.solr.cloud.PeerSyncReplicationTest.bringUpDeadNodeAndEnsureNoReplication(PeerSyncReplicationTest.java:277)
at org.apache.solr.cloud.PeerSyncReplicationTest.forceNodeFailureAndDoPeerSync(PeerSyncReplicationTest.java:259)
at org.apache.solr.cloud.PeerSyncReplicationTest.test(PeerSyncReplicationTest.java:138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 12542 lines...]
[junit4] Suite: org.apache.solr.cloud.PeerSyncReplicationTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/init-core-data-001
[junit4] 2> 1482290 INFO (SUITE-PeerSyncReplicationTest-seed#[FAB2F37EA14E8E10]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields
[junit4] 2> 1482291 INFO (SUITE-PeerSyncReplicationTest-seed#[FAB2F37EA14E8E10]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 1482291 INFO (SUITE-PeerSyncReplicationTest-seed#[FAB2F37EA14E8E10]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /zy_/bp
[junit4] 2> 1482293 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1482293 INFO (Thread-3744) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1482293 INFO (Thread-3744) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1482393 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.ZkTestServer start zk server on port:59058
[junit4] 2> 1482425 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1482426 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 1482427 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1482428 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1482428 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1482429 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1482430 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1482430 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1482431 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1482432 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1482432 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1482528 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/control-001/cores/collection1
[junit4] 2> 1482529 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 1482530 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@729db2f6{/zy_/bp,null,AVAILABLE}
[junit4] 2> 1482531 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@75604273{HTTP/1.1,[http/1.1]}{127.0.0.1:52599}
[junit4] 2> 1482532 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.e.j.s.Server Started @1485585ms
[junit4] 2> 1482532 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/tempDir-001/control/data, hostContext=/zy_/bp, hostPort=52599, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/control-001/cores}
[junit4] 2> 1482532 ERROR (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1482532 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 1482532 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1482532 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1482532 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-11T17:16:36.528Z
[junit4] 2> 1482534 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1482534 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/control-001/solr.xml
[junit4] 2> 1482549 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59058/solr
[junit4] 2> 1482560 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [n:127.0.0.1:52599_zy_%2Fbp ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:52599_zy_%2Fbp
[junit4] 2> 1482560 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [n:127.0.0.1:52599_zy_%2Fbp ] o.a.s.c.Overseer Overseer (id=97441113459261444-127.0.0.1:52599_zy_%2Fbp-n_0000000000) starting
[junit4] 2> 1482577 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [n:127.0.0.1:52599_zy_%2Fbp ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52599_zy_%2Fbp
[junit4] 2> 1482578 INFO (zkCallback-1884-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1482624 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [n:127.0.0.1:52599_zy_%2Fbp ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/control-001/cores
[junit4] 2> 1482624 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [n:127.0.0.1:52599_zy_%2Fbp ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 1482631 INFO (OverseerStateUpdate-97441113459261444-127.0.0.1:52599_zy_%2Fbp-n_0000000000) [n:127.0.0.1:52599_zy_%2Fbp ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1482669 WARN (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 1482670 INFO (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 1482690 INFO (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 1482896 WARN (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 1482898 INFO (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 1482905 INFO (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 1482906 INFO (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/control-001/cores/collection1/data/]
[junit4] 2> 1482906 INFO (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2941199b
[junit4] 2> 1482907 INFO (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=20, maxMergedSegmentMB=18.7861328125, floorSegmentMB=0.8388671875, forceMergeDeletesPctAllowed=17.655500658123156, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 1482950 WARN (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1483032 INFO (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1483032 INFO (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1483033 INFO (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1483033 INFO (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1483034 INFO (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1038527066, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 1483034 INFO (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@505ab4d6[collection1] main]
[junit4] 2> 1483035 INFO (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1483036 INFO (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1483036 INFO (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1483036 INFO (coreLoadExecutor-6201-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559057816126226432
[junit4] 2> 1483049 INFO (coreZkRegister-6194-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp x:collection1 c:control_collection) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1483049 INFO (coreZkRegister-6194-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp x:collection1 c:control_collection) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1483050 INFO (searcherExecutor-6202-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp x:collection1 c:control_collection) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@505ab4d6[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1483062 INFO (coreZkRegister-6194-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp x:collection1 c:control_collection) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:52599/zy_/bp/collection1/
[junit4] 2> 1483062 INFO (coreZkRegister-6194-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp x:collection1 c:control_collection) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1483062 INFO (coreZkRegister-6194-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp x:collection1 c:control_collection) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:52599/zy_/bp/collection1/ has no replicas
[junit4] 2> 1483069 INFO (coreZkRegister-6194-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp x:collection1 c:control_collection) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:52599/zy_/bp/collection1/ shard1
[junit4] 2> 1483219 INFO (coreZkRegister-6194-thread-1-processing-n:127.0.0.1:52599_zy_%2Fbp x:collection1 c:control_collection) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1483435 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1483435 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59058/solr ready
[junit4] 2> 1483435 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1483502 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-1-001/cores/collection1
[junit4] 2> 1483503 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-1-001
[junit4] 2> 1483503 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 1483504 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@55d2012c{/zy_/bp,null,AVAILABLE}
[junit4] 2> 1483504 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@3965e405{HTTP/1.1,[http/1.1]}{127.0.0.1:34585}
[junit4] 2> 1483504 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.e.j.s.Server Started @1486558ms
[junit4] 2> 1483504 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/zy_/bp, hostPort=34585, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-1-001/cores}
[junit4] 2> 1483505 ERROR (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1483505 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 1483505 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1483505 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1483505 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-11T17:16:37.501Z
[junit4] 2> 1483507 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1483507 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-1-001/solr.xml
[junit4] 2> 1483512 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59058/solr
[junit4] 2> 1483518 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1483520 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34585_zy_%2Fbp
[junit4] 2> 1483521 INFO (zkCallback-1884-thread-2-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1483521 INFO (zkCallback-1888-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1483522 INFO (zkCallback-1893-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1483561 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-1-001/cores
[junit4] 2> 1483561 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 1483563 INFO (OverseerStateUpdate-97441113459261444-127.0.0.1:52599_zy_%2Fbp-n_0000000000) [n:127.0.0.1:52599_zy_%2Fbp ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1484593 WARN (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 1484594 INFO (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 1484605 INFO (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 1484682 WARN (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 1484684 INFO (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 1484701 INFO (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1484702 INFO (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-1-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-1-001/cores/collection1/data/]
[junit4] 2> 1484702 INFO (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2941199b
[junit4] 2> 1484703 INFO (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=20, maxMergedSegmentMB=18.7861328125, floorSegmentMB=0.8388671875, forceMergeDeletesPctAllowed=17.655500658123156, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 1484737 WARN (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1484807 INFO (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1484807 INFO (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1484808 INFO (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1484808 INFO (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1484825 INFO (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1038527066, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 1484826 INFO (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@60bb31e5[collection1] main]
[junit4] 2> 1484827 INFO (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1484827 INFO (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1484827 INFO (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1484828 INFO (coreLoadExecutor-6212-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559057818005274624
[junit4] 2> 1484836 INFO (searcherExecutor-6213-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp x:collection1 c:collection1) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@60bb31e5[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1484851 INFO (coreZkRegister-6207-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp x:collection1 c:collection1) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1484851 INFO (coreZkRegister-6207-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp x:collection1 c:collection1) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1484851 INFO (coreZkRegister-6207-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp x:collection1 c:collection1) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:34585/zy_/bp/collection1/
[junit4] 2> 1484851 INFO (coreZkRegister-6207-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp x:collection1 c:collection1) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1484851 INFO (coreZkRegister-6207-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp x:collection1 c:collection1) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:34585/zy_/bp/collection1/ has no replicas
[junit4] 2> 1484870 INFO (coreZkRegister-6207-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp x:collection1 c:collection1) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:34585/zy_/bp/collection1/ shard1
[junit4] 2> 1484972 INFO (coreZkRegister-6207-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp x:collection1 c:collection1) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1485412 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-2-001/cores/collection1
[junit4] 2> 1485413 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-2-001
[junit4] 2> 1485413 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 1485425 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4e44c4f{/zy_/bp,null,AVAILABLE}
[junit4] 2> 1485425 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@3a7f89e6{HTTP/1.1,[http/1.1]}{127.0.0.1:52945}
[junit4] 2> 1485425 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.e.j.s.Server Started @1488479ms
[junit4] 2> 1485425 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/zy_/bp, hostPort=52945, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-2-001/cores}
[junit4] 2> 1485426 ERROR (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1485426 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 1485426 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1485426 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1485426 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-11T17:16:39.422Z
[junit4] 2> 1485467 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1485467 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-2-001/solr.xml
[junit4] 2> 1485472 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59058/solr
[junit4] 2> 1485486 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [n:127.0.0.1:52945_zy_%2Fbp ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 1485489 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [n:127.0.0.1:52945_zy_%2Fbp ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52945_zy_%2Fbp
[junit4] 2> 1485489 INFO (zkCallback-1893-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1485489 INFO (zkCallback-1888-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1485490 INFO (zkCallback-1884-thread-3-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1485495 INFO (zkCallback-1899-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1485543 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [n:127.0.0.1:52945_zy_%2Fbp ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-2-001/cores
[junit4] 2> 1485543 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [n:127.0.0.1:52945_zy_%2Fbp ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 1485544 INFO (OverseerStateUpdate-97441113459261444-127.0.0.1:52599_zy_%2Fbp-n_0000000000) [n:127.0.0.1:52599_zy_%2Fbp ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1486553 WARN (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 1486554 INFO (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 1486564 INFO (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 1486641 WARN (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 1486642 INFO (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 1486650 INFO (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1486651 INFO (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-2-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-2-001/cores/collection1/data/]
[junit4] 2> 1486651 INFO (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2941199b
[junit4] 2> 1486653 INFO (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=20, maxMergedSegmentMB=18.7861328125, floorSegmentMB=0.8388671875, forceMergeDeletesPctAllowed=17.655500658123156, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 1486668 WARN (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1486695 INFO (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1486695 INFO (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1486696 INFO (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1486696 INFO (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1486697 INFO (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1038527066, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 1486697 INFO (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@36fc191a[collection1] main]
[junit4] 2> 1486698 INFO (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1486698 INFO (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1486698 INFO (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1486699 INFO (coreLoadExecutor-6223-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559057819967160320
[junit4] 2> 1486700 INFO (searcherExecutor-6224-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 c:collection1) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@36fc191a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1486703 INFO (coreZkRegister-6218-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 c:collection1) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
[junit4] 2> 1486703 INFO (updateExecutor-1896-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 1486703 INFO (recoveryExecutor-1897-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 1486703 INFO (recoveryExecutor-1897-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 1486703 INFO (recoveryExecutor-1897-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
[junit4] 2> 1486703 INFO (recoveryExecutor-1897-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 1486703 INFO (recoveryExecutor-1897-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:34585/zy_/bp/collection1/] and I am [http://127.0.0.1:52945/zy_/bp/collection1/]
[junit4] 2> 1486705 INFO (recoveryExecutor-1897-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:34585/zy_/bp]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:52945_zy_%252Fbp&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 1486707 INFO (qtp2082678899-12447) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 1486707 INFO (qtp2082678899-12447) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
[junit4] 2> 1486707 INFO (qtp2082678899-12447) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:52945_zy_%2Fbp, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:52945/zy_/bp","node_name":"127.0.0.1:52945_zy_%2Fbp","state":"down"}
[junit4] 2> 1487223 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-3-001/cores/collection1
[junit4] 2> 1487224 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-3-001
[junit4] 2> 1487224 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 1487225 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@e6095f{/zy_/bp,null,AVAILABLE}
[junit4] 2> 1487226 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@7722db97{HTTP/1.1,[http/1.1]}{127.0.0.1:36602}
[junit4] 2> 1487226 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.e.j.s.Server Started @1490280ms
[junit4] 2> 1487226 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/zy_/bp, hostPort=36602, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-3-001/cores}
[junit4] 2> 1487226 ERROR (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1487226 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 1487226 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1487226 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1487226 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-11T17:16:41.222Z
[junit4] 2> 1487234 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1487234 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-3-001/solr.xml
[junit4] 2> 1487239 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59058/solr
[junit4] 2> 1487245 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [n:127.0.0.1:36602_zy_%2Fbp ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 1487247 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [n:127.0.0.1:36602_zy_%2Fbp ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36602_zy_%2Fbp
[junit4] 2> 1487250 INFO (zkCallback-1888-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1487250 INFO (zkCallback-1893-thread-1-processing-n:127.0.0.1:34585_zy_%2Fbp) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1487250 INFO (zkCallback-1899-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp) [n:127.0.0.1:52945_zy_%2Fbp ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1487250 INFO (zkCallback-1884-thread-4-processing-n:127.0.0.1:52599_zy_%2Fbp) [n:127.0.0.1:52599_zy_%2Fbp ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1487258 INFO (zkCallback-1906-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1487344 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [n:127.0.0.1:36602_zy_%2Fbp ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-3-001/cores
[junit4] 2> 1487344 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [n:127.0.0.1:36602_zy_%2Fbp ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 1487345 INFO (OverseerStateUpdate-97441113459261444-127.0.0.1:52599_zy_%2Fbp-n_0000000000) [n:127.0.0.1:52599_zy_%2Fbp ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1487707 INFO (qtp2082678899-12447) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:52945_zy_%2Fbp, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:52945/zy_/bp","node_name":"127.0.0.1:52945_zy_%2Fbp","state":"recovering"}
[junit4] 2> 1487707 INFO (qtp2082678899-12447) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 1487707 INFO (qtp2082678899-12447) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:52945_zy_%252Fbp&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1000
[junit4] 2> 1488208 INFO (recoveryExecutor-1897-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:34585/zy_/bp/collection1/] - recoveringAfterStartup=[true]
[junit4] 2> 1488208 INFO (recoveryExecutor-1897-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=http://127.0.0.1:52945/zy_/bp START replicas=[http://127.0.0.1:34585/zy_/bp/collection1/] nUpdates=1000
[junit4] 2> 1488210 INFO (qtp2082678899-12450) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 1488210 INFO (qtp2082678899-12450) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1] webapp=/zy_/bp path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1488211 INFO (recoveryExecutor-1897-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 1488211 INFO (recoveryExecutor-1897-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
[junit4] 2> 1488211 INFO (recoveryExecutor-1897-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1488211 INFO (recoveryExecutor-1897-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 1488212 INFO (recoveryExecutor-1897-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1488212 INFO (recoveryExecutor-1897-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
[junit4] 2> 1488212 INFO (recoveryExecutor-1897-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
[junit4] 2> 1488212 INFO (recoveryExecutor-1897-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
[junit4] 2> 1488212 INFO (recoveryExecutor-1897-thread-1-processing-n:127.0.0.1:52945_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
[junit4] 2> 1488356 WARN (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 1488357 INFO (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 1488368 INFO (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 1488477 WARN (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 1488479 INFO (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 1488493 INFO (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1488493 INFO (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-3-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001/shard-3-001/cores/collection1/data/]
[junit4] 2> 1488493 INFO (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2941199b
[junit4] 2> 1488495 INFO (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=20, maxMergedSegmentMB=18.7861328125, floorSegmentMB=0.8388671875, forceMergeDeletesPctAllowed=17.655500658123156, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 1488503 WARN (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1488529 INFO (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1488530 INFO (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1488530 INFO (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1488530 INFO (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1488531 INFO (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1038527066, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 1488531 INFO (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7c00ac98[collection1] main]
[junit4] 2> 1488532 INFO (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1488532 INFO (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1488532 INFO (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1488533 INFO (searcherExecutor-6235-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 c:collection1) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7c00ac98[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1488534 INFO (coreLoadExecutor-6234-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559057821891297280
[junit4] 2> 1488537 INFO (coreZkRegister-6229-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 c:collection1) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
[junit4] 2> 1488537 INFO (updateExecutor-1903-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 1488538 INFO (recoveryExecutor-1904-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 1488538 INFO (recoveryExecutor-1904-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 1488538 INFO (recoveryExecutor-1904-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
[junit4] 2> 1488538 INFO (recoveryExecutor-1904-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 1488538 INFO (recoveryExecutor-1904-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:34585/zy_/bp/collection1/] and I am [http://127.0.0.1:36602/zy_/bp/collection1/]
[junit4] 2> 1488540 INFO (recoveryExecutor-1904-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:34585/zy_/bp]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:36602_zy_%252Fbp&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 1488541 INFO (qtp2082678899-12448) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 1488541 INFO (qtp2082678899-12448) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
[junit4] 2> 1488541 INFO (qtp2082678899-12448) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:36602_zy_%2Fbp, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:36602/zy_/bp","node_name":"127.0.0.1:36602_zy_%2Fbp","state":"down"}
[junit4] 2> 1488963 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.SolrTestCaseJ4 ###Starting test
[junit4] 2> 1488963 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Wait for recoveries to finish - wait 30 for each attempt
[junit4] 2> 1488963 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
[junit4] 2> 1489542 INFO (qtp2082678899-12448) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:36602_zy_%2Fbp, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:36602/zy_/bp","node_name":"127.0.0.1:36602_zy_%2Fbp","state":"recovering"}
[junit4] 2> 1489542 INFO (qtp2082678899-12448) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 1489542 INFO (qtp2082678899-12448) [n:127.0.0.1:34585_zy_%2Fbp ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:36602_zy_%252Fbp&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node3&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1000
[junit4] 2> 1490042 INFO (recoveryExecutor-1904-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:34585/zy_/bp/collection1/] - recoveringAfterStartup=[true]
[junit4] 2> 1490043 INFO (recoveryExecutor-1904-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=http://127.0.0.1:36602/zy_/bp START replicas=[http://127.0.0.1:34585/zy_/bp/collection1/] nUpdates=1000
[junit4] 2> 1490044 INFO (qtp2082678899-12451) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 1490044 INFO (qtp2082678899-12451) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1] webapp=/zy_/bp path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1490045 INFO (recoveryExecutor-1904-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 1490046 INFO (recoveryExecutor-1904-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
[junit4] 2> 1490046 INFO (recoveryExecutor-1904-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1490046 INFO (recoveryExecutor-1904-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 1490047 INFO (recoveryExecutor-1904-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1490047 INFO (recoveryExecutor-1904-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
[junit4] 2> 1490047 INFO (recoveryExecutor-1904-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
[junit4] 2> 1490047 INFO (recoveryExecutor-1904-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
[junit4] 2> 1490047 INFO (recoveryExecutor-1904-thread-1-processing-n:127.0.0.1:36602_zy_%2Fbp x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
[junit4] 2> 1490963 INFO (TEST-PeerSyncReplicationTest.test-seed#[FAB2F37EA14E8E10]) [ ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
[junit4] 2> 1490978 INFO (qtp1465612843-12408) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1490978 INFO (qtp1465612843-12408) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 1490980 INFO (qtp1465612843-12408) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1490980 INFO (qtp1465612843-12408) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 2
[junit4] 2> 1490991 INFO (qtp2082678899-12450) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1490991 INFO (qtp2082678899-12450) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 1491010 INFO (qtp2082678899-12450) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1491010 INFO (qtp2082678899-12450) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:34585/zy_/bp/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 19
[junit4] 2> 1491022 INFO (qtp1438273558-12510) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1491023 INFO (qtp1438273558-12510) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 1491023 INFO (qtp1438273558-12510) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1491023 INFO (qtp1639651242-12475) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1491023 INFO (qtp1639651242-12475) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 1491024 INFO (qtp1639651242-12475) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1491024 INFO (qtp1639651242-12475) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:34585/zy_/bp/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 1491024 INFO (qtp1438273558-12510) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:34585/zy_/bp/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 1
[junit4] 2> 1491024 INFO (qtp2082678899-12444) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 42
[junit4] 2> 1491026 INFO (qtp2082678899-12445) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1] webapp=/zy_/bp path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 1491027 INFO (qtp1639651242-12477) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.S.Request [collection1] webapp=/zy_/bp path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 1491028 INFO (qtp1438273558-12508) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.S.Request [collection1] webapp=/zy_/bp path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 1493030 INFO (qtp1465612843-12409) [n:127.0.0.1:52599_zy_%2Fbp c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1559057826604646400)} 0 1
[junit4] 2> 1493035 INFO (qtp1639651242-12476) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={update.distrib=FROMLEADER&_version_=-1559057826606743552&distrib.from=http://127.0.0.1:34585/zy_/bp/collection1/&wt=javabin&version=2}{deleteByQuery=*:* (-1559057826606743552)} 0 1
[junit4] 2> 1493036 INFO (qtp1438273558-12511) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={update.distrib=FROMLEADER&_version_=-1559057826606743552&distrib.from=http://127.0.0.1:34585/zy_/bp/collection1/&wt=javabin&version=2}{deleteByQuery=*:* (-1559057826606743552)} 0 1
[junit4] 2> 1493036 INFO (qtp2082678899-12445) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1559057826606743552)} 0 5
[junit4] 2> 1493038 INFO (qtp1639651242-12479) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:34585/zy_/bp/collection1/&wt=javabin&version=2}{add=[0 (1559057826613035008)]} 0 0
[junit4] 2> 1493048 INFO (qtp1438273558-12513) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:34585/zy_/bp/collection1/&wt=javabin&version=2}{add=[0 (1559057826613035008)]} 0 8
[junit4] 2> 1493048 INFO (qtp2082678899-12451) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={wt=javabin&version=2}{add=[0 (1559057826613035008)]} 0 11
[junit4] 2> 1493049 INFO (qtp1639651242-12479) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:34585/zy_/bp/collection1/&wt=javabin&version=2}{add=[1 (1559057826624569344)]} 0 0
[junit4] 2> 1493049 INFO (qtp1438273558-12514) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:34585/zy_/bp/collection1/&wt=javabin&version=2}{add=[1 (1559057826624569344)]} 0 0
[junit4] 2> 1493049 INFO (qtp2082678899-12449) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={wt=javabin&version=2}{add=[1 (1559057826624569344)]} 0 1
[junit4] 2> 1493050 INFO (qtp1438273558-12507) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:34585/zy_/bp/collection1/&wt=javabin&version=2}{add=[2 (1559057826626666496)]} 0 0
[junit4] 2> 1493051 INFO (qtp1639651242-12475) [n:127.0.0.1:52945_zy_%2Fbp c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:34585/zy_/bp/collection1/&wt=javabin&version=2}{add=[2 (1559057826626666496)]} 0 0
[junit4] 2> 1493055 INFO (qtp2082678899-12450) [n:127.0.0.1:34585_zy_%2Fbp c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={wt=javabin&version=2}{add=[2 (1559057826626666496)]} 0 5
[junit4] 2> 1493056 INFO (qtp1438273558-12507) [n:127.0.0.1:36602_zy_%2Fbp c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/zy_/bp path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:34585/zy_/bp/collectio
[...truncated too long message...]
read-3744) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/aliases.json
[junit4] 2> 6 /solr/clusterprops.json
[junit4] 2> 5 /solr/security.json
[junit4] 2> 5 /solr/configs/conf1
[junit4] 2> 4 /solr/collections/collection1/state.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/clusterstate.json
[junit4] 2> 2 /solr/overseer_elect/election/97441113459261448-127.0.0.1:34585_zy_%2Fbp-n_0000000001
[junit4] 2> 2 /solr/collections/collection1/leader_elect/shard1/election/97441113459261448-core_node1-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 199 /solr/overseer/collection-queue-work
[junit4] 2> 33 /solr/overseer/queue
[junit4] 2> 6 /solr/collections
[junit4] 2> 5 /solr/live_nodes
[junit4] 2> 2 /solr/overseer/queue-work
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=PeerSyncReplicationTest -Dtests.method=test -Dtests.seed=FAB2F37EA14E8E10 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=pt-BR -Dtests.timezone=Canada/Newfoundland -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 199s J0 | PeerSyncReplicationTest.test <<<
[junit4] > Throwable #1: java.lang.AssertionError: timeout waiting to see all nodes active
[junit4] > at __randomizedtesting.SeedInfo.seed([FAB2F37EA14E8E10:72E6CCA40FB2E3E8]:0)
[junit4] > at org.apache.solr.cloud.PeerSyncReplicationTest.waitTillNodesActive(PeerSyncReplicationTest.java:326)
[junit4] > at org.apache.solr.cloud.PeerSyncReplicationTest.bringUpDeadNodeAndEnsureNoReplication(PeerSyncReplicationTest.java:277)
[junit4] > at org.apache.solr.cloud.PeerSyncReplicationTest.forceNodeFailureAndDoPeerSync(PeerSyncReplicationTest.java:259)
[junit4] > at org.apache.solr.cloud.PeerSyncReplicationTest.test(PeerSyncReplicationTest.java:138)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1681709 INFO (SUITE-PeerSyncReplicationTest-seed#[FAB2F37EA14E8E10]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_FAB2F37EA14E8E10-001
[junit4] 2> Feb 11, 2017 5:19:55 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {other_tl1=PostingsFormat(name=MockRandom), range_facet_l_dv=PostingsFormat(name=Direct), rnd_s=PostingsFormat(name=MockRandom), multiDefault=PostingsFormat(name=Asserting), a_t=PostingsFormat(name=Asserting), intDefault=PostingsFormat(name=MockRandom), a_i1=PostingsFormat(name=Asserting), id=PostingsFormat(name=Direct), range_facet_i_dv=PostingsFormat(name=Asserting), text=PostingsFormat(name=LuceneFixedGap), range_facet_l=PostingsFormat(name=Asserting), timestamp=PostingsFormat(name=Asserting)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Lucene70), _version_=DocValuesFormat(name=Memory), range_facet_i_dv=DocValuesFormat(name=Lucene70), intDvoDefault=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1025, maxMBSortInHeap=5.539650777181303, sim=RandomSimilarity(queryNorm=true): {}, locale=pt-BR, timezone=Canada/Newfoundland
[junit4] 2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=4,threads=1,free=142533480,total=531628032
[junit4] 2> NOTE: All tests run in this JVM: [SolrMetricsIntegrationTest, RecoveryAfterSoftCommitTest, BasicAuthIntegrationTest, TestBadConfig, SpatialHeatmapFacetsTest, ForceLeaderTest, OverseerStatusTest, MinimalSchemaTest, TestLRUCache, PolyFieldTest, TestCSVLoader, TestScoreJoinQPNoScore, TestCodecSupport, LeaderElectionIntegrationTest, SyncSliceTest, BlockJoinFacetRandomTest, TestManagedSchema, DistributedDebugComponentTest, BitVectorTest, SolrJmxReporterTest, TestConfigSetsAPIExclusivity, TestTrackingShardHandlerFactory, TestSortByMinMaxFunction, TestJmxIntegration, RuleEngineTest, QueryResultKeyTest, TestAnalyzedSuggestions, CollectionsAPISolrJTest, RankQueryTest, LeaderInitiatedRecoveryOnShardRestartTest, TestConfigSetImmutable, DistributedSpellCheckComponentTest, MultiTermTest, TestRTGBase, TestFunctionQuery, TestAtomicUpdateErrorCases, TestSolrConfigHandler, CoreAdminCreateDiscoverTest, TestInPlaceUpdatesStandalone, QueryElevationComponentTest, InfoHandlerTest, TestNumericTerms64, TestStressLucene, AnalyticsQueryTest, TestSolrQueryParserResource, TestMiniSolrCloudClusterSSL, TriLevelCompositeIdRoutingTest, TestFoldingMultitermQuery, TestFilteredDocIdSet, TestTolerantUpdateProcessorCloud, TestSolrCloudWithHadoopAuthPlugin, SolrCLIZkUtilsTest, PluginInfoTest, ReplicationFactorTest, CursorPagingTest, TestUseDocValuesAsStored, TestSolrCoreProperties, TestReplicationHandlerBackup, TestRandomFaceting, TestManagedSynonymFilterFactory, SSLMigrationTest, TestManagedResourceStorage, TestUnifiedSolrHighlighter, DistributedQueryElevationComponentTest, ClassificationUpdateProcessorIntegrationTest, ReturnFieldsTest, TestLeaderElectionZkExpiry, FastVectorHighlighterTest, TestNRTOpen, ChangedSchemaMergeTest, SignatureUpdateProcessorFactoryTest, RegexBoostProcessorTest, InfixSuggestersTest, TestDistributedSearch, DirectoryFactoryTest, TestQuerySenderListener, ResponseLogComponentTest, HdfsDirectoryTest, TestSolrCloudWithDelegationTokens, TestReRankQParserPlugin, TestMissingGroups, TestPseudoReturnFields, BlockCacheTest, TestOmitPositions, DistributedFacetPivotLargeTest, TestRemoteStreaming, PathHierarchyTokenizerFactoryTest, CdcrRequestHandlerTest, TestFieldResource, AnalysisErrorHandlingTest, TestAuthorizationFramework, TestDistributedStatsComponentCardinality, TestImplicitCoreProperties, ConjunctionSolrSpellCheckerTest, SolrCoreMetricManagerTest, HdfsRecoveryZkTest, TestMacros, BasicZkTest, TestInitQParser, TestSegmentSorting, TestHighlightDedupGrouping, TestJmxMonitoredMap, HdfsWriteToMultipleCollectionsTest, UtilsToolTest, TestSha256AuthenticationProvider, HdfsCollectionsAPIDistributedZkTest, TestFastWriter, CollectionStateFormat2Test, TestRequestForwarding, TestCloudPivotFacet, TestPointFields, TestCrossCoreJoin, TestXmlQParser, TestReplicationHandler, BadIndexSchemaTest, TestFiltering, TestIndexSearcher, SolrIndexSplitterTest, AnalysisAfterCoreReloadTest, TestExtendedDismaxParser, SuggesterFSTTest, SolrRequestParserTest, DocValuesTest, XsltUpdateRequestHandlerTest, DebugComponentTest, DirectSolrSpellCheckerTest, TestQueryTypes, TermsComponentTest, TermVectorComponentTest, TestSolrQueryParser, IndexSchemaTest, UniqFieldsUpdateProcessorFactoryTest, BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, UpdateParamsTest, BadComponentTest, TestMergePolicyConfig, TestSolrDeletionPolicy2, SampleTest, TestDocSet, NumericFieldsTest, TestElisionMultitermQuery, TestFuzzyAnalyzedSuggestions, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, ScriptEngineTest, URLClassifyProcessorTest, TestFastLRUCache, DateMathParserTest, ChaosMonkeyNothingIsSafeTest, CdcrReplicationDistributedZkTest, CdcrReplicationHandlerTest, CdcrVersionReplicationTest, CloudExitableDirectoryReaderTest, CollectionTooManyReplicasTest, DeleteInactiveReplicaTest, DeleteNodeTest, DeleteStatusTest, DistribDocExpirationUpdateProcessorTest, LeaderFailureAfterFreshStartTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, OverseerRolesTest, PeerSyncReplicationTest]
[junit4] Completed [528/694 (1!)] on J0 in 199.95s, 1 test, 1 failure <<< FAILURES!
[...truncated 62696 lines...]
-documentation-lint:
[jtidy] Checking for broken html (such as invalid tags)...
[delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/build/jtidy_tmp
[echo] Checking for broken links...
[exec]
[exec] Crawl/parse...
[exec]
[exec] Verify...
[echo] Checking for malformed docs...
[exec]
[exec] /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/docs/solr-solrj/overview-summary.html
[exec] missing: org.apache.solr.client.solrj.io.eval
[exec]
[exec] Missing javadocs were found!
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:775: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:101: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build.xml:654: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build.xml:670: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/common-build.xml:2493: exec returned: 1
Total time: 82 minutes 6 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-Tests-master - Build # 1657 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/1657/
All tests passed
Build Log:
[...truncated 75224 lines...]
-documentation-lint:
[jtidy] Checking for broken html (such as invalid tags)...
[delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/build/jtidy_tmp
[echo] Checking for broken links...
[exec]
[exec] Crawl/parse...
[exec]
[exec] Verify...
[echo] Checking for malformed docs...
[exec]
[exec] /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/docs/solr-solrj/overview-summary.html
[exec] missing: org.apache.solr.client.solrj.io.eval
[exec]
[exec] Missing javadocs were found!
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:775: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:101: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build.xml:654: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build.xml:670: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/common-build.xml:2493: exec returned: 1
Total time: 80 minutes 43 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-Tests-master - Build # 1656 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/1656/
All tests passed
Build Log:
[...truncated 26262 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:775: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:117: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/common-build.xml:1937: Rat problems were found!
Total time: 73 minutes 18 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-Tests-master - Build # 1655 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/1655/
1 tests failed.
FAILED: org.apache.solr.cloud.PeerSyncReplicationTest.test
Error Message:
timeout waiting to see all nodes active
Stack Trace:
java.lang.AssertionError: timeout waiting to see all nodes active
at __randomizedtesting.SeedInfo.seed([60017695146F4C8C:E855494FBA932174]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.PeerSyncReplicationTest.waitTillNodesActive(PeerSyncReplicationTest.java:326)
at org.apache.solr.cloud.PeerSyncReplicationTest.bringUpDeadNodeAndEnsureNoReplication(PeerSyncReplicationTest.java:277)
at org.apache.solr.cloud.PeerSyncReplicationTest.forceNodeFailureAndDoPeerSync(PeerSyncReplicationTest.java:259)
at org.apache.solr.cloud.PeerSyncReplicationTest.test(PeerSyncReplicationTest.java:138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 13148 lines...]
[junit4] Suite: org.apache.solr.cloud.PeerSyncReplicationTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/init-core-data-001
[junit4] 2> 1529276 INFO (SUITE-PeerSyncReplicationTest-seed#[60017695146F4C8C]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields
[junit4] 2> 1529277 INFO (SUITE-PeerSyncReplicationTest-seed#[60017695146F4C8C]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 1529277 INFO (SUITE-PeerSyncReplicationTest-seed#[60017695146F4C8C]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /qdwcz/qn
[junit4] 2> 1529278 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1529279 INFO (Thread-3490) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1529279 INFO (Thread-3490) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1529379 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.ZkTestServer start zk server on port:50445
[junit4] 2> 1529389 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1529390 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 1529391 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1529392 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1529393 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1529394 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1529395 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1529396 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1529397 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1529398 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1529399 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1529466 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/control-001/cores/collection1
[junit4] 2> 1529468 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 1529468 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6455cfd9{/qdwcz/qn,null,AVAILABLE}
[junit4] 2> 1529470 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@1f0e03ca{SSL,[ssl, http/1.1]}{127.0.0.1:60521}
[junit4] 2> 1529470 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.Server Started @1533122ms
[junit4] 2> 1529470 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/tempDir-001/control/data, hostContext=/qdwcz/qn, hostPort=60521, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/control-001/cores}
[junit4] 2> 1529470 ERROR (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1529471 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.0.0
[junit4] 2> 1529471 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1529471 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1529471 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-10T17:59:05.415Z
[junit4] 2> 1529473 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1529473 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/control-001/solr.xml
[junit4] 2> 1529478 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50445/solr
[junit4] 2> 1529492 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [n:127.0.0.1:60521_qdwcz%2Fqn ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60521_qdwcz%2Fqn
[junit4] 2> 1529493 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [n:127.0.0.1:60521_qdwcz%2Fqn ] o.a.s.c.Overseer Overseer (id=97435618195800069-127.0.0.1:60521_qdwcz%2Fqn-n_0000000000) starting
[junit4] 2> 1529497 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [n:127.0.0.1:60521_qdwcz%2Fqn ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60521_qdwcz%2Fqn
[junit4] 2> 1529510 INFO (zkCallback-1819-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1529596 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [n:127.0.0.1:60521_qdwcz%2Fqn ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/control-001/cores
[junit4] 2> 1529596 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [n:127.0.0.1:60521_qdwcz%2Fqn ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 1529598 INFO (OverseerStateUpdate-97435618195800069-127.0.0.1:60521_qdwcz%2Fqn-n_0000000000) [n:127.0.0.1:60521_qdwcz%2Fqn ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1529615 INFO (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 1529634 INFO (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 1529734 WARN (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 1529735 INFO (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 1529745 INFO (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 1529745 INFO (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/control-001/cores/collection1/data/]
[junit4] 2> 1529745 INFO (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@32b86464
[junit4] 2> 1529747 INFO (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=16, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.23792919836846327]
[junit4] 2> 1529755 WARN (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1529782 INFO (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1529782 INFO (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1529783 INFO (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1529783 INFO (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1529783 INFO (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=33, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7948739787787078]
[junit4] 2> 1529784 INFO (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@396e0254[collection1] main]
[junit4] 2> 1529785 INFO (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1529785 INFO (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1529785 INFO (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1529786 INFO (coreLoadExecutor-5673-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1558969891663380480
[junit4] 2> 1529789 INFO (searcherExecutor-5674-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn x:collection1 c:control_collection) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@396e0254[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1529808 INFO (coreZkRegister-5666-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn x:collection1 c:control_collection) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1529808 INFO (coreZkRegister-5666-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn x:collection1 c:control_collection) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1529808 INFO (coreZkRegister-5666-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn x:collection1 c:control_collection) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:60521/qdwcz/qn/collection1/
[junit4] 2> 1529808 INFO (coreZkRegister-5666-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn x:collection1 c:control_collection) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1529808 INFO (coreZkRegister-5666-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn x:collection1 c:control_collection) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:60521/qdwcz/qn/collection1/ has no replicas
[junit4] 2> 1529827 INFO (coreZkRegister-5666-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn x:collection1 c:control_collection) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:60521/qdwcz/qn/collection1/ shard1
[junit4] 2> 1529992 INFO (coreZkRegister-5666-thread-1-processing-n:127.0.0.1:60521_qdwcz%2Fqn x:collection1 c:control_collection) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1530079 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1530079 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:50445/solr ready
[junit4] 2> 1530079 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1530079 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
[junit4] 2> 1530148 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-1-001/cores/collection1
[junit4] 2> 1530149 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-1-001
[junit4] 2> 1530150 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 1530151 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@55540b62{/qdwcz/qn,null,AVAILABLE}
[junit4] 2> 1530152 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@54650116{SSL,[ssl, http/1.1]}{127.0.0.1:56973}
[junit4] 2> 1530154 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.Server Started @1533806ms
[junit4] 2> 1530154 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/qdwcz/qn, hostPort=56973, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-1-001/cores}
[junit4] 2> 1530154 ERROR (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1530154 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.0.0
[junit4] 2> 1530154 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1530154 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1530154 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-10T17:59:06.098Z
[junit4] 2> 1530156 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1530156 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-1-001/solr.xml
[junit4] 2> 1530162 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50445/solr
[junit4] 2> 1530167 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1530169 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56973_qdwcz%2Fqn
[junit4] 2> 1530170 INFO (zkCallback-1823-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1530170 INFO (zkCallback-1819-thread-4-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1530172 INFO (zkCallback-1829-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1530212 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-1-001/cores
[junit4] 2> 1530212 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 1530214 INFO (OverseerStateUpdate-97435618195800069-127.0.0.1:60521_qdwcz%2Fqn-n_0000000000) [n:127.0.0.1:60521_qdwcz%2Fqn ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1530316 WARN (OverseerStateUpdate-97435618195800069-127.0.0.1:60521_qdwcz%2Fqn-n_0000000000) [n:127.0.0.1:60521_qdwcz%2Fqn ] o.a.s.c.Overseer Bad version writing to ZK using compare-and-set, will force refresh cluster state: KeeperErrorCode = BadVersion for /collections/collection1/state.json
[junit4] 2> 1530320 INFO (zkCallback-1829-thread-2-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1530321 INFO (OverseerStateUpdate-97435618195800069-127.0.0.1:60521_qdwcz%2Fqn-n_0000000000) [n:127.0.0.1:60521_qdwcz%2Fqn ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1530321 INFO (zkCallback-1829-thread-2-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1531224 INFO (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 1531235 INFO (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 1531326 WARN (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 1531328 INFO (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 1531336 INFO (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1531336 INFO (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-1-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-1-001/cores/collection1/data/]
[junit4] 2> 1531336 INFO (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@32b86464
[junit4] 2> 1531338 INFO (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=16, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.23792919836846327]
[junit4] 2> 1531346 WARN (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1531370 INFO (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1531370 INFO (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1531371 INFO (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1531371 INFO (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1531372 INFO (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=33, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7948739787787078]
[junit4] 2> 1531372 INFO (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@44c89c0e[collection1] main]
[junit4] 2> 1531373 INFO (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1531373 INFO (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1531373 INFO (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1531375 INFO (searcherExecutor-5685-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn x:collection1 c:collection1) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@44c89c0e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1531375 INFO (coreLoadExecutor-5684-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1558969893329567744
[junit4] 2> 1531380 INFO (coreZkRegister-5679-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn x:collection1 c:collection1) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1531381 INFO (coreZkRegister-5679-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn x:collection1 c:collection1) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1531381 INFO (coreZkRegister-5679-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn x:collection1 c:collection1) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:56973/qdwcz/qn/collection1/
[junit4] 2> 1531381 INFO (coreZkRegister-5679-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn x:collection1 c:collection1) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1531381 INFO (coreZkRegister-5679-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn x:collection1 c:collection1) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:56973/qdwcz/qn/collection1/ has no replicas
[junit4] 2> 1531385 INFO (coreZkRegister-5679-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn x:collection1 c:collection1) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:56973/qdwcz/qn/collection1/ shard1
[junit4] 2> 1531488 INFO (zkCallback-1829-thread-2-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1531537 INFO (coreZkRegister-5679-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn x:collection1 c:collection1) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1531648 INFO (zkCallback-1829-thread-2-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1531685 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-2-001/cores/collection1
[junit4] 2> 1531686 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-2-001
[junit4] 2> 1531687 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 1531688 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@71248d1e{/qdwcz/qn,null,AVAILABLE}
[junit4] 2> 1531690 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@93c8780{SSL,[ssl, http/1.1]}{127.0.0.1:35003}
[junit4] 2> 1531691 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.Server Started @1535343ms
[junit4] 2> 1531691 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/qdwcz/qn, hostPort=35003, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-2-001/cores}
[junit4] 2> 1531691 ERROR (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1531691 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.0.0
[junit4] 2> 1531691 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1531691 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1531691 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-10T17:59:07.635Z
[junit4] 2> 1531693 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1531693 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-2-001/solr.xml
[junit4] 2> 1531699 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50445/solr
[junit4] 2> 1531706 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [n:127.0.0.1:35003_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 1531713 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [n:127.0.0.1:35003_qdwcz%2Fqn ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35003_qdwcz%2Fqn
[junit4] 2> 1531715 INFO (zkCallback-1823-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1531715 INFO (zkCallback-1829-thread-2-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1531715 INFO (zkCallback-1819-thread-2-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1531723 INFO (zkCallback-1835-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1531767 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [n:127.0.0.1:35003_qdwcz%2Fqn ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-2-001/cores
[junit4] 2> 1531767 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [n:127.0.0.1:35003_qdwcz%2Fqn ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 1531773 INFO (OverseerStateUpdate-97435618195800069-127.0.0.1:60521_qdwcz%2Fqn-n_0000000000) [n:127.0.0.1:60521_qdwcz%2Fqn ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1531875 INFO (zkCallback-1829-thread-2-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1531875 INFO (zkCallback-1835-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1532781 INFO (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 1532792 INFO (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 1532873 WARN (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 1532875 INFO (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 1532885 INFO (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1532885 INFO (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-2-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-2-001/cores/collection1/data/]
[junit4] 2> 1532885 INFO (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@32b86464
[junit4] 2> 1532886 INFO (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=16, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.23792919836846327]
[junit4] 2> 1532895 WARN (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1532953 INFO (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1532953 INFO (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1532954 INFO (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1532954 INFO (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1532955 INFO (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=33, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7948739787787078]
[junit4] 2> 1532956 INFO (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4f1468ce[collection1] main]
[junit4] 2> 1532957 INFO (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1532958 INFO (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1532958 INFO (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1532959 INFO (coreLoadExecutor-5695-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1558969894990512128
[junit4] 2> 1532962 INFO (searcherExecutor-5696-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 c:collection1) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4f1468ce[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1532970 INFO (coreZkRegister-5690-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 c:collection1) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
[junit4] 2> 1532970 INFO (updateExecutor-1832-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 1532970 INFO (recoveryExecutor-1833-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 1532971 INFO (recoveryExecutor-1833-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 1532971 INFO (recoveryExecutor-1833-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
[junit4] 2> 1532971 INFO (recoveryExecutor-1833-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 1532971 INFO (recoveryExecutor-1833-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [https://127.0.0.1:56973/qdwcz/qn/collection1/] and I am [https://127.0.0.1:35003/qdwcz/qn/collection1/]
[junit4] 2> 1532973 INFO (recoveryExecutor-1833-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [https://127.0.0.1:56973/qdwcz/qn]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:35003_qdwcz%252Fqn&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 1532997 INFO (qtp202009668-12496) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 1532997 INFO (qtp202009668-12496) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
[junit4] 2> 1532997 INFO (qtp202009668-12496) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:35003_qdwcz%2Fqn, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"https://127.0.0.1:35003/qdwcz/qn","node_name":"127.0.0.1:35003_qdwcz%2Fqn","state":"down"}
[junit4] 2> 1533074 INFO (zkCallback-1835-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1533074 INFO (zkCallback-1829-thread-2-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1533245 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-3-001/cores/collection1
[junit4] 2> 1533246 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-3-001
[junit4] 2> 1533250 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 1533251 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@b0e698c{/qdwcz/qn,null,AVAILABLE}
[junit4] 2> 1533253 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@2847274c{SSL,[ssl, http/1.1]}{127.0.0.1:44688}
[junit4] 2> 1533253 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.Server Started @1536905ms
[junit4] 2> 1533253 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/qdwcz/qn, hostPort=44688, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-3-001/cores}
[junit4] 2> 1533254 ERROR (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1533254 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.0.0
[junit4] 2> 1533254 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1533254 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1533254 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-02-10T17:59:09.198Z
[junit4] 2> 1533263 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1533263 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-3-001/solr.xml
[junit4] 2> 1533268 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50445/solr
[junit4] 2> 1533279 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [n:127.0.0.1:44688_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 1533282 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [n:127.0.0.1:44688_qdwcz%2Fqn ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44688_qdwcz%2Fqn
[junit4] 2> 1533283 INFO (zkCallback-1823-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1533283 INFO (zkCallback-1829-thread-2-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1533283 INFO (zkCallback-1835-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1533284 INFO (zkCallback-1819-thread-3-processing-n:127.0.0.1:60521_qdwcz%2Fqn) [n:127.0.0.1:60521_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1533289 INFO (zkCallback-1842-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1533345 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [n:127.0.0.1:44688_qdwcz%2Fqn ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-3-001/cores
[junit4] 2> 1533345 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [n:127.0.0.1:44688_qdwcz%2Fqn ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 1533347 INFO (OverseerStateUpdate-97435618195800069-127.0.0.1:60521_qdwcz%2Fqn-n_0000000000) [n:127.0.0.1:60521_qdwcz%2Fqn ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1533448 INFO (zkCallback-1835-thread-2-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1533448 INFO (zkCallback-1842-thread-2-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1533448 INFO (zkCallback-1829-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1533997 INFO (qtp202009668-12496) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:35003_qdwcz%2Fqn, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"https://127.0.0.1:35003/qdwcz/qn","node_name":"127.0.0.1:35003_qdwcz%2Fqn","state":"recovering"}
[junit4] 2> 1533998 INFO (qtp202009668-12496) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 1533998 INFO (qtp202009668-12496) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:35003_qdwcz%252Fqn&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1000
[junit4] 2> 1534355 INFO (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 1534366 INFO (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 1534451 WARN (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 1534453 INFO (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 1534461 INFO (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1534461 INFO (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-3-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001/shard-3-001/cores/collection1/data/]
[junit4] 2> 1534461 INFO (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@32b86464
[junit4] 2> 1534463 INFO (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=16, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.23792919836846327]
[junit4] 2> 1534471 WARN (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1534498 INFO (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1534499 INFO (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1534499 INFO (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1534499 INFO (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1534500 INFO (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=33, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7948739787787078]
[junit4] 2> 1534500 INFO (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@20b5414[collection1] main]
[junit4] 2> 1534501 INFO (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1534501 INFO (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1534501 INFO (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1534502 INFO (searcherExecutor-5707-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 c:collection1) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@20b5414[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1534503 INFO (coreLoadExecutor-5706-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1558969896609513472
[junit4] 2> 1534506 INFO (coreZkRegister-5701-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 c:collection1) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
[junit4] 2> 1534506 INFO (updateExecutor-1839-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 1534506 INFO (recoveryExecutor-1840-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 1534507 INFO (recoveryExecutor-1840-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 1534507 INFO (recoveryExecutor-1840-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
[junit4] 2> 1534507 INFO (recoveryExecutor-1840-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 1534507 INFO (recoveryExecutor-1840-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [https://127.0.0.1:56973/qdwcz/qn/collection1/] and I am [https://127.0.0.1:44688/qdwcz/qn/collection1/]
[junit4] 2> 1534509 INFO (recoveryExecutor-1840-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [https://127.0.0.1:56973/qdwcz/qn]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:44688_qdwcz%252Fqn&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 1534512 INFO (qtp202009668-12495) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 1534513 INFO (qtp202009668-12495) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
[junit4] 2> 1534513 INFO (qtp202009668-12495) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:44688_qdwcz%2Fqn, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"https://127.0.0.1:44688/qdwcz/qn","node_name":"127.0.0.1:44688_qdwcz%2Fqn","state":"down"}
[junit4] 2> 1534610 INFO (zkCallback-1842-thread-2-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1534610 INFO (zkCallback-1829-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1534610 INFO (zkCallback-1835-thread-2-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1534744 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.SolrTestCaseJ4 ###Starting test
[junit4] 2> 1534744 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Wait for recoveries to finish - wait 30 for each attempt
[junit4] 2> 1534744 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
[junit4] 2> 1535513 INFO (qtp202009668-12495) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:44688_qdwcz%2Fqn, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"https://127.0.0.1:44688/qdwcz/qn","node_name":"127.0.0.1:44688_qdwcz%2Fqn","state":"recovering"}
[junit4] 2> 1535513 INFO (qtp202009668-12495) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 1535513 INFO (qtp202009668-12495) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:44688_qdwcz%252Fqn&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node3&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1000
[junit4] 2> 1540998 INFO (recoveryExecutor-1833-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [https://127.0.0.1:56973/qdwcz/qn/collection1/] - recoveringAfterStartup=[true]
[junit4] 2> 1540998 INFO (recoveryExecutor-1833-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=https://127.0.0.1:35003/qdwcz/qn START replicas=[https://127.0.0.1:56973/qdwcz/qn/collection1/] nUpdates=1000
[junit4] 2> 1541002 INFO (qtp202009668-12496) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 1541002 INFO (qtp202009668-12496) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1] webapp=/qdwcz/qn path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1541002 INFO (recoveryExecutor-1833-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 1541002 INFO (recoveryExecutor-1833-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
[junit4] 2> 1541003 INFO (recoveryExecutor-1833-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1541003 INFO (recoveryExecutor-1833-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 1541003 INFO (recoveryExecutor-1833-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1541003 INFO (recoveryExecutor-1833-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
[junit4] 2> 1541003 INFO (recoveryExecutor-1833-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
[junit4] 2> 1541003 INFO (recoveryExecutor-1833-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
[junit4] 2> 1541003 INFO (recoveryExecutor-1833-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
[junit4] 2> 1541005 INFO (zkCallback-1829-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1541005 INFO (zkCallback-1842-thread-2-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1541005 INFO (zkCallback-1835-thread-2-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1542514 INFO (recoveryExecutor-1840-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [https://127.0.0.1:56973/qdwcz/qn/collection1/] - recoveringAfterStartup=[true]
[junit4] 2> 1542514 INFO (recoveryExecutor-1840-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=https://127.0.0.1:44688/qdwcz/qn START replicas=[https://127.0.0.1:56973/qdwcz/qn/collection1/] nUpdates=1000
[junit4] 2> 1542519 INFO (qtp202009668-12494) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 1542519 INFO (qtp202009668-12494) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1] webapp=/qdwcz/qn path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1542520 INFO (recoveryExecutor-1840-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 1542520 INFO (recoveryExecutor-1840-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
[junit4] 2> 1542520 INFO (recoveryExecutor-1840-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1542520 INFO (recoveryExecutor-1840-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 1542520 INFO (recoveryExecutor-1840-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1542520 INFO (recoveryExecutor-1840-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
[junit4] 2> 1542520 INFO (recoveryExecutor-1840-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
[junit4] 2> 1542520 INFO (recoveryExecutor-1840-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
[junit4] 2> 1542520 INFO (recoveryExecutor-1840-thread-1-processing-n:127.0.0.1:44688_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:44688_qdwcz%2Fqn c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
[junit4] 2> 1542522 INFO (zkCallback-1835-thread-2-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1542522 INFO (zkCallback-1829-thread-1-processing-n:127.0.0.1:56973_qdwcz%2Fqn) [n:127.0.0.1:56973_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1542522 INFO (zkCallback-1842-thread-2-processing-n:127.0.0.1:44688_qdwcz%2Fqn) [n:127.0.0.1:44688_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1542752 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
[junit4] 2> 1542775 INFO (qtp1433608486-12453) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1542775 INFO (qtp1433608486-12453) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 1542775 INFO (qtp1433608486-12453) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1542775 INFO (qtp1433608486-12453) [n:127.0.0.1:60521_qdwcz%2Fqn c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/qdwcz/qn path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 0
[junit4] 2> 1542786 INFO (qtp202009668-12495) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1542786 INFO (qtp202009668-12495) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 1542786 INFO (qtp202009668-12495) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1542786 INFO (qtp202009668-12495) [n:127.0.0.1:56973_qdwcz%2Fqn c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/qdwcz/qn path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:56973/qdwcz/qn/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 1542806 INFO (qtp1842187250-12525) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1542806 INFO (qtp1842187250-12525) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 1542806 INFO (qtp1842187250-12525) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1542806 INFO (qtp1842187250-12525) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/qdwcz/qn path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:56973/qdwcz/qn/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=fa
[...truncated too long message...]
node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection1] coreNodeName=[core_node2]
[junit4] 2> 1733151 INFO (recoveryExecutor-1847-thread-1-processing-n:127.0.0.1:35003_qdwcz%2Fqn x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
[junit4] 2> 1733152 WARN (updateExecutor-1846-thread-2-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Skipping recovery because Solr is shutdown
[junit4] 2> 1733152 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.Overseer Overseer (id=97435618195800083-127.0.0.1:35003_qdwcz%2Fqn-n_0000000004) closing
[junit4] 2> 1733152 INFO (OverseerStateUpdate-97435618195800083-127.0.0.1:35003_qdwcz%2Fqn-n_0000000004) [n:127.0.0.1:35003_qdwcz%2Fqn ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:35003_qdwcz%2Fqn
[junit4] 2> 1733163 WARN (zkCallback-1849-thread-2-processing-n:127.0.0.1:35003_qdwcz%2Fqn) [n:127.0.0.1:35003_qdwcz%2Fqn ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 1733164 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
[junit4] 2> 1733168 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@c2ea926{SSL,[ssl, http/1.1]}{127.0.0.1:35003}
[junit4] 2> 1733168 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@150752ee{/qdwcz/qn,null,UNAVAILABLE}
[junit4] 2> 1733174 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.ChaosMonkey monkey: stop shard! 44688
[junit4] 2> 1733186 INFO (TEST-PeerSyncReplicationTest.test-seed#[60017695146F4C8C]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:50445 50445
[junit4] 2> 1733268 INFO (Thread-3490) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:50445 50445
[junit4] 2> 1733269 WARN (Thread-3490) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/aliases.json
[junit4] 2> 5 /solr/security.json
[junit4] 2> 5 /solr/configs/conf1
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/clusterstate.json
[junit4] 2> 6 /solr/clusterprops.json
[junit4] 2> 4 /solr/collections/collection1/state.json
[junit4] 2> 2 /solr/collections/collection1/leader_elect/shard1/election/97435618195800074-core_node1-n_0000000000
[junit4] 2> 2 /solr/overseer_elect/election/97435618195800074-127.0.0.1:56973_qdwcz%2Fqn-n_0000000001
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 205 /solr/overseer/collection-queue-work
[junit4] 2> 31 /solr/overseer/queue
[junit4] 2> 13 /solr/overseer/queue-work
[junit4] 2> 6 /solr/collections
[junit4] 2> 5 /solr/live_nodes
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=PeerSyncReplicationTest -Dtests.method=test -Dtests.seed=60017695146F4C8C -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=ar-DZ -Dtests.timezone=America/Porto_Acre -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 204s J0 | PeerSyncReplicationTest.test <<<
[junit4] > Throwable #1: java.lang.AssertionError: timeout waiting to see all nodes active
[junit4] > at __randomizedtesting.SeedInfo.seed([60017695146F4C8C:E855494FBA932174]:0)
[junit4] > at org.apache.solr.cloud.PeerSyncReplicationTest.waitTillNodesActive(PeerSyncReplicationTest.java:326)
[junit4] > at org.apache.solr.cloud.PeerSyncReplicationTest.bringUpDeadNodeAndEnsureNoReplication(PeerSyncReplicationTest.java:277)
[junit4] > at org.apache.solr.cloud.PeerSyncReplicationTest.forceNodeFailureAndDoPeerSync(PeerSyncReplicationTest.java:259)
[junit4] > at org.apache.solr.cloud.PeerSyncReplicationTest.test(PeerSyncReplicationTest.java:138)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1733271 INFO (SUITE-PeerSyncReplicationTest-seed#[60017695146F4C8C]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_60017695146F4C8C-001
[junit4] 2> Feb 10, 2017 6:02:29 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=false): {}, locale=ar-DZ, timezone=America/Porto_Acre
[junit4] 2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=4,threads=1,free=258375640,total=529530880
[junit4] 2> NOTE: All tests run in this JVM: [ForceLeaderTest, HdfsRecoveryZkTest, TestSearcherReuse, ConnectionManagerTest, TestFieldCacheWithThreads, ClusterStateTest, ReplicationFactorTest, DeleteInactiveReplicaTest, TestAuthenticationFramework, OverseerCollectionConfigSetProcessorTest, SolrCoreTest, TestLockTree, TestRandomDVFaceting, TestSolrCloudWithHadoopAuthPlugin, PeerSyncWithIndexFingerprintCachingTest, TestNumericTerms64, SpellPossibilityIteratorTest, TestAtomicUpdateErrorCases, TestReplicationHandlerBackup, TestFieldResource, DistanceUnitsTest, TestSolrCloudSnapshots, MinimalSchemaTest, TestRTimerTree, TestMiniSolrCloudClusterSSL, ClassificationUpdateProcessorIntegrationTest, TestSolrConfigHandler, TestSubQueryTransformerCrossCore, TestComplexPhraseQParserPlugin, TestSolrQueryParserResource, TestCrossCoreJoin, TestMissingGroups, TestNamedUpdateProcessors, TestConfigSetsAPIExclusivity, TestStressCloudBlindAtomicUpdates, PeerSyncTest, HdfsBasicDistributedZkTest, TestPointFields, TestSegmentSorting, TestInitQParser, AnalysisErrorHandlingTest, PolyFieldTest, TestDynamicFieldCollectionResource, PluginInfoTest, TestGeoJSONResponseWriter, TestDistributedSearch, TestSolr4Spatial2, TestBlobHandler, ScriptEngineTest, TestManagedSynonymFilterFactory, AssignTest, SSLMigrationTest, AnalyticsQueryTest, TestSolrConfigHandlerConcurrent, SpatialRPTFieldTypeTest, MetricUtilsTest, NotRequiredUniqueKeyTest, RecoveryAfterSoftCommitTest, HLLSerializationTest, SolrCLIZkUtilsTest, DistributedDebugComponentTest, DirectoryFactoryTest, TestSuggestSpellingConverter, RankQueryTest, SolrCoreMetricManagerTest, TestQueryWrapperFilter, ZkStateReaderTest, ResponseLogComponentTest, CloudMLTQParserTest, SignatureUpdateProcessorFactoryTest, ConjunctionSolrSpellCheckerTest, TestNRTOpen, TestScoreJoinQPNoScore, SyncSliceTest, DocExpirationUpdateProcessorFactoryTest, TestFastWriter, TestAuthorizationFramework, TestJmxIntegration, TestRandomRequestDistribution, ChangedSchemaMergeTest, InfoHandlerTest, TestCloudPivotFacet, StandardRequestHandlerTest, TestFaceting, PreAnalyzedUpdateProcessorTest, TestPathTrie, TestPushWriter, TestLazyCores, DistributedFacetPivotLargeTest, ZkNodePropsTest, TestSolrQueryResponse, TestClusterProperties, BasicDistributedZkTest, BitVectorTest, QueryResultKeyTest, TestFileDictionaryLookup, LeaderFailoverAfterPartitionTest, TestCorePropertiesReload, TestDistributedStatsComponentCardinality, TestXmlQParser, TestInPlaceUpdatesStandalone, TestRandomFaceting, TestTrackingShardHandlerFactory, UUIDFieldTest, TestUseDocValuesAsStored, TestMacros, DistributedFacetPivotSmallTest, TestImplicitCoreProperties, TestQuerySenderListener, TestLMJelinekMercerSimilarityFactory, TestFieldCache, DocumentAnalysisRequestHandlerTest, ShardRoutingTest, TestReplicationHandler, HardAutoCommitTest, SimpleFacetsTest, TestSolr4Spatial, BadIndexSchemaTest, ConvertedLegacyTest, TestFunctionQuery, BasicFunctionalityTest, DirectUpdateHandlerTest, SoftAutoCommitTest, TestQueryUtils, PrimitiveFieldTypeTest, TestOmitPositions, RequestHandlersTest, PathHierarchyTokenizerFactoryTest, RequiredFieldsTest, FastVectorHighlighterTest, SolrPluginUtilsTest, ReturnFieldsTest, TestAnalyzedSuggestions, BinaryUpdateRequestHandlerTest, TestLFUCache, CdcrBootstrapTest, ConcurrentDeleteAndCreateCollectionTest, DeleteLastCustomShardedReplicaTest, DeleteReplicaTest, DistributedQueueTest, DistributedVersionInfoTest, LeaderFailureAfterFreshStartTest, OverseerModifyCollectionTest, OverseerRolesTest, OverseerStatusTest, PeerSyncReplicationTest]
[junit4] Completed [547/694 (1!)] on J0 in 205.02s, 1 test, 1 failure <<< FAILURES!
[...truncated 13122 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:775: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:117: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/common-build.xml:1937: Rat problems were found!
Total time: 71 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-Tests-master - Build # 1654 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/1654/
All tests passed
Build Log:
[...truncated 26248 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:775: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:117: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/common-build.xml:1937: Rat problems were found!
Total time: 72 minutes 54 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any