You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Adrien Grand <jp...@gmail.com> on 2018/05/28 14:34:51 UTC
Re: lucene-solr:branch_7x: SOLR-12358: Autoscaling suggestions fail
randomly with sorting
Hi Noble,
This commit shouldn't add a 8.0.0 section to the 7.x changelog, can you
fix? It looks like there are other unintended changes to the changelog.
Le lun. 28 mai 2018 à 08:30, <no...@apache.org> a écrit :
> Repository: lucene-solr
> Updated Branches:
> refs/heads/branch_7x dc0dc1d6e -> a875300a8
>
>
> SOLR-12358: Autoscaling suggestions fail randomly with sorting
>
>
> Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
> Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/a875300a
> Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/a875300a
> Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/a875300a
>
> Branch: refs/heads/branch_7x
> Commit: a875300a897521bc618d5072b20fcd60c8f13985
> Parents: dc0dc1d
> Author: Noble Paul <no...@apache.org>
> Authored: Thu May 24 01:26:50 2018 +1000
> Committer: Noble Paul <no...@apache.org>
> Committed: Mon May 28 16:30:23 2018 +1000
>
> ----------------------------------------------------------------------
> solr/CHANGES.txt | 130 ++++----
> .../autoscaling/AutoScalingHandlerTest.java | 4 +-
> .../client/solrj/cloud/autoscaling/Policy.java | 32 +-
> .../solrj/cloud/autoscaling/PolicyHelper.java | 1 +
> .../solrj/cloud/autoscaling/Preference.java | 3 -
> .../client/solrj/cloud/autoscaling/Row.java | 10 +-
> .../solrj/cloud/autoscaling/TestPolicy.java | 310 ++++++++++++++++++-
> 7 files changed, 422 insertions(+), 68 deletions(-)
> ----------------------------------------------------------------------
>
>
>
> http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a875300a/solr/CHANGES.txt
> ----------------------------------------------------------------------
> diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
> index 2537d37..99ff4b8 100644
> --- a/solr/CHANGES.txt
> +++ b/solr/CHANGES.txt
> @@ -16,6 +16,35 @@ In this release, there is an example Solr server
> including a bundled
> servlet container in the directory named "example".
> See the Solr tutorial at
> https://lucene.apache.org/solr/guide/solr-tutorial.html
>
> +================== 8.0.0 ==================
> +
> +Consult the LUCENE_CHANGES.txt file for additional, low level, changes in
> this release.
> +
> +Versions of Major Components
> +---------------------
> +Apache Tika 1.16
> +Carrot2 3.15.0
> +Velocity 1.7 and Velocity Tools 2.0
> +Apache UIMA 2.3.1
> +Apache ZooKeeper 3.4.11
> +Jetty 9.4.10.v20180503
> +
> +Upgrade Notes
> +----------------------
> +
> +* LUCENE-7996: The 'func' query parser now returns scores that are equal
> to 0
> + when a negative value is produced. This change is due to the fact that
> + Lucene now requires scores to be positive. (Adrien Grand)
> +
> +* SOLR-11882: SolrMetric registries retained references to SolrCores when
> closed. A
> + change of SolrMetricMAnager.registerGauge and
> SolrMetricProducer.initializeMetrics
> + method signatures was required to fix it. Third party components that
> use this API
> + need to be updated. (Eros Taborelli, Erick Erickson, ab)
> +
> +* LUCENE-8267: Memory codecs have been removed from the codebase
> (MemoryPostings,
> + MemoryDocValues). If you used postingsFormat="Memory" or
> docValuesFormat="Memory"
> + switch to "Direct" instead. (Dawid Weiss)
> +
> ================== 7.4.0 ==================
>
> Consult the LUCENE_CHANGES.txt file for additional, low level, changes in
> this release.
> @@ -23,11 +52,11 @@ Consult the LUCENE_CHANGES.txt file for additional,
> low level, changes in this r
> Versions of Major Components
> ---------------------
> Apache Tika 1.17
> -Carrot2 3.16.0
> +Carrot2 3.15.0
> Velocity 1.7 and Velocity Tools 2.0
> Apache UIMA 2.3.1
> Apache ZooKeeper 3.4.11
> -Jetty 9.3.20.v20170531
> +Jetty 9.4.10.v20180503
>
> Upgrade Notes
> ----------------------
> @@ -44,8 +73,6 @@ Upgrade Notes
> New Features
> ----------------------
>
> -* SOLR-12396: Upgrade Carrot2 to 3.16.0, HPPC to 0.8.1, morfologik to
> 2.1.5. (Dawid Weiss)
> -
> * SOLR-11200: A new CMS config option 'ioThrottle' to manually
> enable/disable
> ConcurrentMergeSchedule.doAutoIOThrottle. (Amrit Sarkar, Nawab Zada
> Asad iqbal via Dawid Weiss)
>
> @@ -92,29 +119,9 @@ New Features
> * SOLR-9480: A new 'relatedness()' aggregate function for JSON Faceting
> to enable building Semantic
> Knowledge Graphs. (Trey Grainger, hossman)
>
> -* SOLR-12378: Support missing versionField on indexed docs in
> DocBasedVersionConstraintsURP.
> - (Oliver Bates, Michael Braun via Mark Miller)
> -
> -* SOLR-12388: Enable a strict ZooKeeper-connected search request mode, in
> which search
> - requests will fail when the coordinating node can't communicate with
> ZooKeeper,
> - by setting the "shards.tolerant" param to "requireZkConnected". (Steve
> Rowe)
> -
> -* SOLR-9685: #Tagging queries in JSON Query DSL, equivalent to
> LocalParams based query/filter
> - tagging. Multiple tags are comma separated.
> - LocalParams Example : {!tag=colorfilt}color:blue
> - Equivalent JSON Example : { "#colorfilt" : "color:blue" }
> - (Dmitry Tikhonov, Mikhail Khludnev, yonik)
> -
> -* SOLR-12328: JSON Facet API: Domain change with graph query.
> - (Daniel Meehl, Kevin Watters, yonik)
> -
> Bug Fixes
> ----------------------
>
> -* SOLR-5351: Fixed More Like This Handler to use all fields provided in
> mlt.fl when used with
> - content stream. The similarity is calculated between the content
> stream's value and all
> - fields listed in mlt.fl. (Dawid Weiss)
> -
> * SOLR-12103: Raise CryptoKeys.DEFAULT_KEYPAIR_LENGTH from 1024 to 2048.
> (Mark Miller)
>
> * SOLR-12107: Fixed a error in [child] transformer that could ocur if
> documentCache was not used (hossman)
> @@ -142,12 +149,7 @@ Bug Fixes
>
> * SOLR-11929: UpdateLog metrics are not initialized on core reload. (ab,
> Steve Rowe)
>
> -* SOLR-11882: SolrMetric registries retained references to SolrCores when
> closed. A
> - change of SolrMetricMAnager.registerGauge and
> SolrMetricProducer.initializeMetrics
> - method signatures was required to fix it. Third party components may
> continue to use the old API
> - but should be updated to avoid this bug (Eros Taborelli, Erick
> Erickson, ab)
> -
> -* SOLR-12199: TestReplicationHandler.doTestRepeater(): TEST_PORT
> interpolation failure:
> +* SOLR-12199: TestReplicationHandler.doTestRepeater(): TEST_PORT
> interpolation failure:
> Server refused connection at: http://127.0.0.1:TEST_PORT/solr
> (Mikhail Khludnev, Dawid Weiss, Steve Rowe)
>
> * SOLR-12096: Fixed inconsistent results format of subquery transformer
> for distributed search (multi-shard).
> @@ -200,6 +202,9 @@ Bug Fixes
> * SOLR-12284: WordBreakSolrSpellchecker will no longer add parenthesis in
> collations when breaking words in
> non-boolean queries. (James Dyer)
>
> +* SOLR-12290: Do not close any servlet streams and improve our servlet
> stream closing prevention code for users
> + and devs. (Mark Miller)
> +
> * SOLR-12293: Updates need to use their own connection pool to maintain
> connection reuse and prevent spurious
> recoveries. (Mark Miller)
>
> @@ -231,6 +236,8 @@ Bug Fixes
> * SOLR-12294: update processors loaded from runtime jars fail to load if
> they are specified
> in an update processor chain (noble)
>
> +* SOLR-12358: Autoscaling suggestions fail randomly with sorting (noble)
> +
> Optimizations
> ----------------------
>
> @@ -252,11 +259,6 @@ Optimizations
> * SOLR-11880: Avoid creating new exceptions for every request made to
> MDCAwareThreadPoolExecutor by distributed
> search and update operations. (Varun Thacker, shalin)
>
> -* SOLR-12375: Optimize Lucene needsScore / ScoreMode use:
> - A non-cached filter query could be told incorrectly that scores were
> needed.
> - The /export (ExportQParserPlugin) would declare incorrectly that scores
> are needed.
> - Expanded docs (expand component) could be told incorrectly that scores
> are needed. (David Smiley)
> -
> Other Changes
> ----------------------
>
> @@ -412,7 +414,6 @@ Upgrade Notes
>
> New Features
> ----------------------
> -
> * SOLR-11285: Simulation framework for autoscaling. (ab)
>
> * LUCENE-2899: In the Solr analysis-extras contrib, added support for the
> @@ -806,9 +807,9 @@ New Features
>
> * SOLR-11524: A new autoscaling/suggestions API end-point which gives
> autoscaling suggestions (noble)
>
> -* SOLR-11519: Implement suggestions for replica count violations (noble)
> +* SOLR-11519: Implement autoscaling suggestions for replica count
> violations (noble)
>
> -* SOLR-11518: Implement Suggestions for freedisk violations (noble)
> +* SOLR-11518: Implement autoscaling Suggestions for freedisk violations
> (noble)
>
> * SOLR-10132: A new optional facet.matches parameter to return facet
> buckets only
> for terms that match a regular expression. (Gus Heck, Christine
> Poerschke)
> @@ -902,6 +903,9 @@ Bug Fixes
>
> * SOLR-11413: SolrGraphiteReporter fails to report metrics due to
> non-thread safe code. (Erik Persson, ab)
>
> +* SOLR-9440: The ZkStateReader.removeCollectionStateWatcher method can
> cache a DocCollection reference and
> + never update it causing stale state to be returned in ClusterState.
> (shalin)
> +
> * SOLR-11560: Specifying the replicationFactor parameter while restoring
> a collection would lead to extra tlog
> and pull replicas being created (Peter Szantai-Kis, Varun Thacker)
>
> @@ -909,9 +913,6 @@ Bug Fixes
>
> * SOLR-11503: Collections created with legacyCloud=true cannot be opened
> if legacyCloud=false (Erick Erickson)
>
> -* SOLR-9440: The ZkStateReader.removeCollectionStateWatcher method can
> cache a DocCollection reference and
> - never update it causing stale state to be returned in ClusterState.
> (shalin)
> -
> * SOLR-11619: V2 requests that needed to be forwarded to other nodes
> would get an NPE. (David Smiley)
>
> * SOLR-11231: Guard against unset fields when performing language
> detection.
> @@ -924,6 +925,18 @@ Bug Fixes
> admin UI dashboard would not display any commandline arguments.
> (Webster Homer via Shawn Heisey)
>
> +* SOLR-11608: Correctly parse the new core-name in the V2 core rename API.
> + (Jason Gerlowski via Anshum Gupta)
> +
> +* SOLR-11256: The queue size for ConcurrentUpdateSolrClient should
> default to 10 instead of throwing an
> + IllegalArgumentException. (Jason Gerlowski, Anshum Gupta)
> +
> +* SOLR-11616: Snapshot the segments more robustly such that segments
> created during a backup does does not fail the
> + operation (Varun Thacker)
> +
> +* SOLR-11687: SolrCore.getNewIndexDir falsely returns {dataDir}/index on
> any IOException reading index.properties
> + (Nikolay Martynov, Erick Erickson)
> +
> * SOLR-9137: bin/solr script ignored custom STOP_PORT on shutdown.
> (Joachim Kohlhammer, Steve Rowe, Christine Poerschke)
>
> @@ -938,18 +951,6 @@ Bug Fixes
> * SOLR-11691: V2 requests for create-alias didn't work when the
> collections param was an array.
> (Jason Gerlowski, Gus Heck, David Smiley, noble)
>
> -* SOLR-11608: Correctly parse the new core-name in the V2 core rename API.
> - (Jason Gerlowski via Anshum Gupta)
> -
> -* SOLR-11256: The queue size for ConcurrentUpdateSolrClient should
> default to 10 instead of throwing an
> - IllegalArgumentException. (Jason Gerlowski, Anshum Gupta)
> -
> -* SOLR-11616: Snapshot the segments more robustly such that segments
> created during a backup does does not fail the
> - operation (Varun Thacker)
> -
> -* SOLR-11687: SolrCore.getNewIndexDir falsely returns {dataDir}/index on
> any IOException reading index.properties
> - (Nikolay Martynov, Erick Erickson)
> -
> Optimizations
> ----------------------
> * SOLR-11285: Refactor autoscaling framework to avoid direct references
> to Zookeeper and Solr
> @@ -1965,12 +1966,12 @@ Other Changes
> * SOLR-11068: MOVEREPLICA and REPLACENODE API parameter names are now
> 'sourceNode' and 'targetNode'. The old names
> viz. 'fromNode' for MOVEREPLICA and 'source', 'target' for REPLACENODE
> have been deprecated. (shalin)
>
> +* SOLR-11088: Fix sporadic failures of
> MetricsHandlerTest.testPropertyFilter on jenkins (shalin)
> +
> * SOLR-11037: Refactor to provide NodeConfig.getSolrDataHome internal
> API. (ab, janhoy, shalin)
>
> * SOLR-11119: Switch from Trie to Points field types in the .system
> collection schema. (Steve Rowe)
>
> -* SOLR-11088: Fix sporadic failures of
> MetricsHandlerTest.testPropertyFilter on jenkins (shalin)
> -
> * SOLR-10494: Make default response format JSON (wt=json), and also
> indent text responses formats
> (indent=on) by default (Trey Grainger & Cassandra Targett via hossman)
>
> @@ -2070,6 +2071,25 @@ Other Changes
>
> * SOLR-11324: Clean up mention of trie fields in documentation and source
> comments. (Steve Rowe)
>
> +================== 6.6.4 ==================
> +
> +Consult the LUCENE_CHANGES.txt file for additional, low level, changes in
> this release.
> +
> +Versions of Major Components
> +---------------------
> +Apache Tika 1.17
> +Carrot2 3.15.0
> +Velocity 1.7 and Velocity Tools 2.0
> +Apache UIMA 2.3.1
> +Apache ZooKeeper 3.4.11
> +Jetty 9.4.10.v20180503
> +
> +Bug Fixes
> +----------------------
> +
> +* SOLR-12316: Do not allow to use absolute URIs for including other files
> in solrconfig.xml and schema parsing.
> + (Ananthesh, Ishan Chattopadhyaya, Uwe Schindler)
> +
> ================== 6.6.3 ==================
>
> Consult the LUCENE_CHANGES.txt file for additional, low level, changes in
> this release.
>
>
> http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a875300a/solr/core/src/test/org/apache/solr/cloud/autoscaling/AutoScalingHandlerTest.java
> ----------------------------------------------------------------------
> diff --git
> a/solr/core/src/test/org/apache/solr/cloud/autoscaling/AutoScalingHandlerTest.java
> b/solr/core/src/test/org/apache/solr/cloud/autoscaling/AutoScalingHandlerTest.java
> index 2aec88e..483b60c 100644
> ---
> a/solr/core/src/test/org/apache/solr/cloud/autoscaling/AutoScalingHandlerTest.java
> +++
> b/solr/core/src/test/org/apache/solr/cloud/autoscaling/AutoScalingHandlerTest.java
> @@ -757,14 +757,14 @@ public class AutoScalingHandlerTest extends
> SolrCloudTestCase {
> response = solrClient.request(req);
>
> Map<String, Object> diagnostics = (Map<String, Object>)
> response.get("diagnostics");
> - List sortedNodes = (List) diagnostics.get("sortedNodes");
> + List sortedNodes = (List) Utils.getObjectByPath(response, false,
> "diagnostics/sortedNodes");
> assertNotNull(sortedNodes);
>
> assertEquals(2, sortedNodes.size());
> for (int i = 0; i < 2; i++) {
> Map node = (Map) sortedNodes.get(i);
> assertNotNull(node);
> - assertEquals(5, node.size());
> + assertEquals(6, node.size());
> assertNotNull(node.get("node"));
> assertNotNull(node.get("cores"));
> assertEquals(0L, node.get("cores"));
>
>
> http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a875300a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
> ----------------------------------------------------------------------
> diff --git
> a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
> b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
> index 05c9c20..fb01cc5 100644
> ---
> a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
> +++
> b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
> @@ -25,6 +25,7 @@ import java.util.Collection;
> import java.util.Collections;
> import java.util.HashMap;
> import java.util.HashSet;
> +import java.util.Iterator;
> import java.util.LinkedHashMap;
> import java.util.List;
> import java.util.Map;
> @@ -359,14 +360,35 @@ public class Policy implements MapWriter {
> }
>
> static void setApproxValuesAndSortNodes(List<Preference>
> clusterPreferences, List<Row> matrix) {
> + List<Row> deadNodes = null;
> + Iterator<Row> it =matrix.iterator();
> + while (it.hasNext()){
> + Row row = it.next();
> + if(!row.isLive){
> + if(deadNodes == null) deadNodes = new ArrayList<>();
> + deadNodes.add(row);
> + it.remove();
> + }
> + }
> +
> if (!clusterPreferences.isEmpty()) {
> //this is to set the approximate value according to the precision
> ArrayList<Row> tmpMatrix = new ArrayList<>(matrix);
> + Row[] lastComparison = new Row[2];
> for (Preference p : clusterPreferences) {
> try {
> - tmpMatrix.sort((r1, r2) -> p.compare(r1, r2, false));
> + tmpMatrix.sort((r1, r2) -> {
> + lastComparison[0] = r1;
> + lastComparison[1] = r2;
> + return p.compare(r1, r2, false);
> + });
> } catch (Exception e) {
> - LOG.error("Exception! prefs = {}, matrix = {}",
> clusterPreferences, matrix);
> + LOG.error("Exception! prefs = {}, recent r1 = {}, r2 = {},
> compare : {} matrix = {}",
> + clusterPreferences,
> + lastComparison[0].node,
> + lastComparison[1].node,
> + p.compare(lastComparison[0],lastComparison[1], false ),
> + Utils.toJSONString(Utils.getDeepCopy(tmpMatrix, 6, false)));
> throw e;
> }
> p.setApproxVal(tmpMatrix);
> @@ -378,6 +400,12 @@ public class Policy implements MapWriter {
> if (result == 0) result = clusterPreferences.get(0).compare(r1,
> r2, false);
> return result;
> });
> +
> + if(deadNodes != null){
> + for (Row deadNode : deadNodes) {
> + matrix.add(0, deadNode);
> + }
> + }
> }
> }
>
>
>
> http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a875300a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/PolicyHelper.java
> ----------------------------------------------------------------------
> diff --git
> a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/PolicyHelper.java
> b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/PolicyHelper.java
> index 11c43d5..112ae76 100644
> ---
> a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/PolicyHelper.java
> +++
> b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/PolicyHelper.java
> @@ -190,6 +190,7 @@ public class PolicyHelper {
> List<Map<String, Object>> sortedNodes = new
> ArrayList<>(sorted.size());
> for (Row row : sorted) {
> Map<String, Object> map = Utils.makeMap("node", row.node);
> + map.put("isLive", row.isLive);
> for (Cell cell : row.getCells()) {
> for (Preference clusterPreference : clusterPreferences) {
> Policy.SortParam name = clusterPreference.getName();
>
>
> http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a875300a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Preference.java
> ----------------------------------------------------------------------
> diff --git
> a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Preference.java
> b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Preference.java
> index 3e73632..0db9886 100644
> ---
> a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Preference.java
> +++
> b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Preference.java
> @@ -63,9 +63,6 @@ public class Preference implements MapWriter {
> // recursive, it uses the precision to tie & when there is a tie use
> the next preference to compare
> // in non-recursive mode, precision is not taken into consideration and
> sort is done on actual value
> int compare(Row r1, Row r2, boolean useApprox) {
> - if (!r1.isLive && !r2.isLive) return 0;
> - if (!r1.isLive) return -1;
> - if (!r2.isLive) return 1;
> Object o1 = useApprox ? r1.cells[idx].approxVal : r1.cells[idx].val;
> Object o2 = useApprox ? r2.cells[idx].approxVal : r2.cells[idx].val;
> int result = 0;
>
>
> http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a875300a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Row.java
> ----------------------------------------------------------------------
> diff --git
> a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Row.java
> b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Row.java
> index 659f315..f1799c4 100644
> ---
> a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Row.java
> +++
> b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Row.java
> @@ -19,6 +19,7 @@ package org.apache.solr.client.solrj.cloud.autoscaling;
>
> import java.io.IOException;
> import java.util.ArrayList;
> +import java.util.Arrays;
> import java.util.Collections;
> import java.util.HashMap;
> import java.util.List;
> @@ -27,7 +28,6 @@ import java.util.Random;
> import java.util.function.Consumer;
> import java.util.stream.Collectors;
>
> -import org.apache.solr.common.IteratorWriter;
> import org.apache.solr.common.MapWriter;
> import org.apache.solr.common.cloud.Replica;
> import org.apache.solr.common.cloud.ZkStateReader;
> @@ -81,10 +81,10 @@ public class Row implements MapWriter {
>
> @Override
> public void writeMap(EntryWriter ew) throws IOException {
> - ew.put(node, (IteratorWriter) iw -> {
> - iw.add((MapWriter) e -> e.put("replicas",
> collectionVsShardVsReplicas));
> - for (Cell cell : cells) iw.add(cell);
> - });
> + ew.put(NODE, node);
> + ew.put("replicas", collectionVsShardVsReplicas);
> + ew.put("isLive", isLive);
> + ew.put("attributes", Arrays.asList(cells));
> }
>
> Row copy(Policy.Session session) {
>
>
> http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a875300a/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
> ----------------------------------------------------------------------
> diff --git
> a/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
> b/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
> index a53b60c..a099171 100644
> ---
> a/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
> +++
> b/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
> @@ -2160,5 +2160,313 @@ public void testUtilizeNodeFailure2() throws
> Exception {
> assertEquals("count = "+count ,1,count);
> }
>
> -
> + //SOLR-12358
> + public void testSortError() {
> + Policy policy = new Policy((Map<String, Object>)
> Utils.fromJSONString("{cluster-preferences: [{minimize : cores,
> precision:1}, " +
> + "{maximize : freedisk, precision: 50}, " +
> + "{minimize: sysLoadAvg}]}"));
> + String rowsData = "{'sortedNodes':[" +
> + " {" +
> + " 'node':'solr-01:8983_solr'," +
> + " 'replicas':{}," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':2}," +
> + " {'freedisk':1734.5261459350586}," +
> + " {'sysLoadAvg':35.0}," +
> + " {'node':'solr-01:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-07:8983_solr'," +
> + " 'replicas':{}," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1721.5669250488281}," +
> + " {'sysLoadAvg':10.0}," +
> + " {'node':'solr-07:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-08:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1764.9518203735352}," +
> + " {'sysLoadAvg':330.0}," +
> + " {'node':'solr-08:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-25:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1779.7792778015137}," +
> + " {'sysLoadAvg':304.0}," +
> + " {'node':'solr-25:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-15:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1697.5930519104004}," +
> + " {'sysLoadAvg':277.0}," +
> + " {'node':'solr-15:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-13:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':2}," +
> + " {'freedisk':1755.1909484863281}," +
> + " {'sysLoadAvg':265.0}," +
> + " {'node':'solr-13:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-14:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1757.6035423278809}," +
> + " {'sysLoadAvg':61.0}," +
> + " {'node':'solr-14:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-16:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1746.081386566162}," +
> + " {'sysLoadAvg':260.0}," +
> + " {'node':'solr-16:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-04:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':2}," +
> + " {'freedisk':1708.7230529785156}," +
> + " {'sysLoadAvg':216.0}," +
> + " {'node':'solr-04:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-06:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1688.3182678222656}," +
> + " {'sysLoadAvg':385.0}," +
> + " {'node':'solr-06:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-02:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':6}," +
> + " {'freedisk':1778.226963043213}," +
> + " {'sysLoadAvg':369.0}," +
> + " {'node':'solr-02:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-05:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1741.9401931762695}," +
> + " {'sysLoadAvg':354.0}," +
> + " {'node':'solr-05:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-23:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1718.854579925537}," +
> + " {'sysLoadAvg':329.0}," +
> + " {'node':'solr-23:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-24:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1733.6669311523438}," +
> + " {'sysLoadAvg':327.0}," +
> + " {'node':'solr-24:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-09:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1714.6191711425781}," +
> + " {'sysLoadAvg':278.0}," +
> + " {'node':'solr-09:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-10:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1755.3038482666016}," +
> + " {'sysLoadAvg':266.0}," +
> + " {'node':'solr-10:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-28:8983_solr'," +
> + " 'isLive':false," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1691.3830909729004}," +
> + " {'sysLoadAvg':261.0}," +
> + " {'node':'solr-28:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-29:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':2}," +
> + " {'freedisk':1706.797966003418}," +
> + " {'sysLoadAvg':252.99999999999997}," +
> + " {'node':'solr-29:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-32:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1762.432300567627}," +
> + " {'sysLoadAvg':221.0}," +
> + " {'node':'solr-32:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-21:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1760.9801979064941}," +
> + " {'sysLoadAvg':213.0}," +
> + " {'node':'solr-21:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-22:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1780.5297241210938}," +
> + " {'sysLoadAvg':209.0}," +
> + " {'node':'solr-22:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-31:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1700.1481628417969}," +
> + " {'sysLoadAvg':211.0}," +
> + " {'node':'solr-31:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-33:8983_solr'," +
> + " 'isLive':false," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1748.1132926940918}," +
> + " {'sysLoadAvg':199.0}," +
> + " {'node':'solr-33:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-36:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1776.197639465332}," +
> + " {'sysLoadAvg':193.0}," +
> + " {'node':'solr-36:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-35:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1746.7729606628418}," +
> + " {'sysLoadAvg':191.0}," +
> + " {'node':'solr-35:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-12:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1713.287540435791}," +
> + " {'sysLoadAvg':175.0}," +
> + " {'node':'solr-12:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-11:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1736.784511566162}," +
> + " {'sysLoadAvg':169.0}," +
> + " {'node':'solr-11:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-35:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1766.9416885375977}," +
> + " {'sysLoadAvg':155.0}," +
> + " {'node':'solr-35:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-17:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1764.3425407409668}," +
> + " {'sysLoadAvg':139.0}," +
> + " {'node':'solr-17:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-18:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':2}," +
> + " {'freedisk':1757.0613975524902}," +
> + " {'sysLoadAvg':132.0}," +
> + " {'node':'solr-18:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-20:8983_solr'," +
> + " 'isLive':false," +
> + " 'attributes':[" +
> + " {'cores':3}," +
> + " {'freedisk':1747.4205322265625}," +
> + " {'sysLoadAvg':126.0}," +
> + " {'node':'solr-20:8983_solr'}]}," +
> + " {" +
> + " 'node':'solr-27:8983_solr'," +
> + " 'isLive':true," +
> + " 'attributes':[" +
> + " {'cores':4}," +
> + " {'freedisk':1721.0442085266113}," +
> + " {'sysLoadAvg':118.0}," +
> + " {'node':'solr-27:8983_solr'}]}]}";
> +
> + List l = (List) ((Map)
> Utils.fromJSONString(rowsData)).get("sortedNodes");
> + List<Suggestion.ConditionType> params = new ArrayList<>();
> + params.add(Suggestion.ConditionType.CORES);
> + params.add(Suggestion.ConditionType.FREEDISK);
> + params.add(Suggestion.ConditionType.SYSLOADAVG);
> + params.add(Suggestion.ConditionType.NODE);
> + List<Row> rows = new ArrayList<>();
> + for (Object o : l) {
> + Map m = (Map) o;
> + Cell[] c = new Cell[params.size()];
> + List attrs = (List) m.get("attributes");
> + for (int i = 0; i < params.size(); i++) {
> + Suggestion.ConditionType param = params.get(i);
> + for (Object attr : attrs) {
> + Object o1 = ((Map) attr).get(param.tagName);
> + if (o1 != null) {
> + o1 = param.validate(param.tagName, o1, false);
> + c[i] = new Cell(i, param.tagName, o1, o1, param, null);
> + }
> + }
> + }
> + rows.add(new Row((String) m.get("node"), c, false,
> + new HashMap<>(),
> + (Boolean) m.get("isLive"), null));
> + }
> + int deadNodes = 0;
> + for (Row row : rows) {
> + if (!row.isLive) deadNodes++;
> + }
> +
> + Policy.setApproxValuesAndSortNodes(policy.clusterPreferences, rows);
> +
> + for (int i = 0; i < deadNodes; i++) {
> + assertFalse(rows.get(i).isLive);
> + }
> +
> + for (int i = deadNodes; i < rows.size(); i++) {
> + assertTrue(rows.get(i).isLive);
> + }
> +
> +
> + }
> +
> +
> }
>
>
Re: lucene-solr:branch_7x: SOLR-12358: Autoscaling suggestions fail
randomly with sorting
Posted by Adrien Grand <jp...@gmail.com>.
I just removed the 8.0 section to fix the smoketester build, but I think it
needs other changes as well, for instance the fact that it changed the
version of the components looks wrong to me.
Le lun. 28 mai 2018 à 16:34, Adrien Grand <jp...@gmail.com> a écrit :
> Hi Noble,
>
> This commit shouldn't add a 8.0.0 section to the 7.x changelog, can you
> fix? It looks like there are other unintended changes to the changelog.
>
> Le lun. 28 mai 2018 à 08:30, <no...@apache.org> a écrit :
>
>> Repository: lucene-solr
>> Updated Branches:
>> refs/heads/branch_7x dc0dc1d6e -> a875300a8
>>
>>
>> SOLR-12358: Autoscaling suggestions fail randomly with sorting
>>
>>
>> Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
>> Commit:
>> http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/a875300a
>> Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/a875300a
>> Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/a875300a
>>
>> Branch: refs/heads/branch_7x
>> Commit: a875300a897521bc618d5072b20fcd60c8f13985
>> Parents: dc0dc1d
>> Author: Noble Paul <no...@apache.org>
>> Authored: Thu May 24 01:26:50 2018 +1000
>> Committer: Noble Paul <no...@apache.org>
>> Committed: Mon May 28 16:30:23 2018 +1000
>>
>> ----------------------------------------------------------------------
>> solr/CHANGES.txt | 130 ++++----
>> .../autoscaling/AutoScalingHandlerTest.java | 4 +-
>> .../client/solrj/cloud/autoscaling/Policy.java | 32 +-
>> .../solrj/cloud/autoscaling/PolicyHelper.java | 1 +
>> .../solrj/cloud/autoscaling/Preference.java | 3 -
>> .../client/solrj/cloud/autoscaling/Row.java | 10 +-
>> .../solrj/cloud/autoscaling/TestPolicy.java | 310 ++++++++++++++++++-
>> 7 files changed, 422 insertions(+), 68 deletions(-)
>> ----------------------------------------------------------------------
>>
>>
>>
>> http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a875300a/solr/CHANGES.txt
>> ----------------------------------------------------------------------
>> diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
>> index 2537d37..99ff4b8 100644
>> --- a/solr/CHANGES.txt
>> +++ b/solr/CHANGES.txt
>> @@ -16,6 +16,35 @@ In this release, there is an example Solr server
>> including a bundled
>> servlet container in the directory named "example".
>> See the Solr tutorial at
>> https://lucene.apache.org/solr/guide/solr-tutorial.html
>>
>> +================== 8.0.0 ==================
>> +
>> +Consult the LUCENE_CHANGES.txt file for additional, low level, changes
>> in this release.
>> +
>> +Versions of Major Components
>> +---------------------
>> +Apache Tika 1.16
>> +Carrot2 3.15.0
>> +Velocity 1.7 and Velocity Tools 2.0
>> +Apache UIMA 2.3.1
>> +Apache ZooKeeper 3.4.11
>> +Jetty 9.4.10.v20180503
>> +
>> +Upgrade Notes
>> +----------------------
>> +
>> +* LUCENE-7996: The 'func' query parser now returns scores that are equal
>> to 0
>> + when a negative value is produced. This change is due to the fact that
>> + Lucene now requires scores to be positive. (Adrien Grand)
>> +
>> +* SOLR-11882: SolrMetric registries retained references to SolrCores
>> when closed. A
>> + change of SolrMetricMAnager.registerGauge and
>> SolrMetricProducer.initializeMetrics
>> + method signatures was required to fix it. Third party components that
>> use this API
>> + need to be updated. (Eros Taborelli, Erick Erickson, ab)
>> +
>> +* LUCENE-8267: Memory codecs have been removed from the codebase
>> (MemoryPostings,
>> + MemoryDocValues). If you used postingsFormat="Memory" or
>> docValuesFormat="Memory"
>> + switch to "Direct" instead. (Dawid Weiss)
>> +
>> ================== 7.4.0 ==================
>>
>> Consult the LUCENE_CHANGES.txt file for additional, low level, changes
>> in this release.
>> @@ -23,11 +52,11 @@ Consult the LUCENE_CHANGES.txt file for additional,
>> low level, changes in this r
>> Versions of Major Components
>> ---------------------
>> Apache Tika 1.17
>> -Carrot2 3.16.0
>> +Carrot2 3.15.0
>> Velocity 1.7 and Velocity Tools 2.0
>> Apache UIMA 2.3.1
>> Apache ZooKeeper 3.4.11
>> -Jetty 9.3.20.v20170531
>> +Jetty 9.4.10.v20180503
>>
>> Upgrade Notes
>> ----------------------
>> @@ -44,8 +73,6 @@ Upgrade Notes
>> New Features
>> ----------------------
>>
>> -* SOLR-12396: Upgrade Carrot2 to 3.16.0, HPPC to 0.8.1, morfologik to
>> 2.1.5. (Dawid Weiss)
>> -
>> * SOLR-11200: A new CMS config option 'ioThrottle' to manually
>> enable/disable
>> ConcurrentMergeSchedule.doAutoIOThrottle. (Amrit Sarkar, Nawab Zada
>> Asad iqbal via Dawid Weiss)
>>
>> @@ -92,29 +119,9 @@ New Features
>> * SOLR-9480: A new 'relatedness()' aggregate function for JSON Faceting
>> to enable building Semantic
>> Knowledge Graphs. (Trey Grainger, hossman)
>>
>> -* SOLR-12378: Support missing versionField on indexed docs in
>> DocBasedVersionConstraintsURP.
>> - (Oliver Bates, Michael Braun via Mark Miller)
>> -
>> -* SOLR-12388: Enable a strict ZooKeeper-connected search request mode,
>> in which search
>> - requests will fail when the coordinating node can't communicate with
>> ZooKeeper,
>> - by setting the "shards.tolerant" param to "requireZkConnected".
>> (Steve Rowe)
>> -
>> -* SOLR-9685: #Tagging queries in JSON Query DSL, equivalent to
>> LocalParams based query/filter
>> - tagging. Multiple tags are comma separated.
>> - LocalParams Example : {!tag=colorfilt}color:blue
>> - Equivalent JSON Example : { "#colorfilt" : "color:blue" }
>> - (Dmitry Tikhonov, Mikhail Khludnev, yonik)
>> -
>> -* SOLR-12328: JSON Facet API: Domain change with graph query.
>> - (Daniel Meehl, Kevin Watters, yonik)
>> -
>> Bug Fixes
>> ----------------------
>>
>> -* SOLR-5351: Fixed More Like This Handler to use all fields provided in
>> mlt.fl when used with
>> - content stream. The similarity is calculated between the content
>> stream's value and all
>> - fields listed in mlt.fl. (Dawid Weiss)
>> -
>> * SOLR-12103: Raise CryptoKeys.DEFAULT_KEYPAIR_LENGTH from 1024 to 2048.
>> (Mark Miller)
>>
>> * SOLR-12107: Fixed a error in [child] transformer that could ocur if
>> documentCache was not used (hossman)
>> @@ -142,12 +149,7 @@ Bug Fixes
>>
>> * SOLR-11929: UpdateLog metrics are not initialized on core reload.
>> (ab, Steve Rowe)
>>
>> -* SOLR-11882: SolrMetric registries retained references to SolrCores
>> when closed. A
>> - change of SolrMetricMAnager.registerGauge and
>> SolrMetricProducer.initializeMetrics
>> - method signatures was required to fix it. Third party components may
>> continue to use the old API
>> - but should be updated to avoid this bug (Eros Taborelli, Erick
>> Erickson, ab)
>> -
>> -* SOLR-12199: TestReplicationHandler.doTestRepeater(): TEST_PORT
>> interpolation failure:
>> +* SOLR-12199: TestReplicationHandler.doTestRepeater(): TEST_PORT
>> interpolation failure:
>> Server refused connection at: http://127.0.0.1:TEST_PORT/solr
>> (Mikhail Khludnev, Dawid Weiss, Steve Rowe)
>>
>> * SOLR-12096: Fixed inconsistent results format of subquery transformer
>> for distributed search (multi-shard).
>> @@ -200,6 +202,9 @@ Bug Fixes
>> * SOLR-12284: WordBreakSolrSpellchecker will no longer add parenthesis
>> in collations when breaking words in
>> non-boolean queries. (James Dyer)
>>
>> +* SOLR-12290: Do not close any servlet streams and improve our servlet
>> stream closing prevention code for users
>> + and devs. (Mark Miller)
>> +
>> * SOLR-12293: Updates need to use their own connection pool to maintain
>> connection reuse and prevent spurious
>> recoveries. (Mark Miller)
>>
>> @@ -231,6 +236,8 @@ Bug Fixes
>> * SOLR-12294: update processors loaded from runtime jars fail to load if
>> they are specified
>> in an update processor chain (noble)
>>
>> +* SOLR-12358: Autoscaling suggestions fail randomly with sorting (noble)
>> +
>> Optimizations
>> ----------------------
>>
>> @@ -252,11 +259,6 @@ Optimizations
>> * SOLR-11880: Avoid creating new exceptions for every request made to
>> MDCAwareThreadPoolExecutor by distributed
>> search and update operations. (Varun Thacker, shalin)
>>
>> -* SOLR-12375: Optimize Lucene needsScore / ScoreMode use:
>> - A non-cached filter query could be told incorrectly that scores were
>> needed.
>> - The /export (ExportQParserPlugin) would declare incorrectly that
>> scores are needed.
>> - Expanded docs (expand component) could be told incorrectly that scores
>> are needed. (David Smiley)
>> -
>> Other Changes
>> ----------------------
>>
>> @@ -412,7 +414,6 @@ Upgrade Notes
>>
>> New Features
>> ----------------------
>> -
>> * SOLR-11285: Simulation framework for autoscaling. (ab)
>>
>> * LUCENE-2899: In the Solr analysis-extras contrib, added support for the
>> @@ -806,9 +807,9 @@ New Features
>>
>> * SOLR-11524: A new autoscaling/suggestions API end-point which gives
>> autoscaling suggestions (noble)
>>
>> -* SOLR-11519: Implement suggestions for replica count violations (noble)
>> +* SOLR-11519: Implement autoscaling suggestions for replica count
>> violations (noble)
>>
>> -* SOLR-11518: Implement Suggestions for freedisk violations (noble)
>> +* SOLR-11518: Implement autoscaling Suggestions for freedisk violations
>> (noble)
>>
>> * SOLR-10132: A new optional facet.matches parameter to return facet
>> buckets only
>> for terms that match a regular expression. (Gus Heck, Christine
>> Poerschke)
>> @@ -902,6 +903,9 @@ Bug Fixes
>>
>> * SOLR-11413: SolrGraphiteReporter fails to report metrics due to
>> non-thread safe code. (Erik Persson, ab)
>>
>> +* SOLR-9440: The ZkStateReader.removeCollectionStateWatcher method can
>> cache a DocCollection reference and
>> + never update it causing stale state to be returned in ClusterState.
>> (shalin)
>> +
>> * SOLR-11560: Specifying the replicationFactor parameter while restoring
>> a collection would lead to extra tlog
>> and pull replicas being created (Peter Szantai-Kis, Varun Thacker)
>>
>> @@ -909,9 +913,6 @@ Bug Fixes
>>
>> * SOLR-11503: Collections created with legacyCloud=true cannot be opened
>> if legacyCloud=false (Erick Erickson)
>>
>> -* SOLR-9440: The ZkStateReader.removeCollectionStateWatcher method can
>> cache a DocCollection reference and
>> - never update it causing stale state to be returned in ClusterState.
>> (shalin)
>> -
>> * SOLR-11619: V2 requests that needed to be forwarded to other nodes
>> would get an NPE. (David Smiley)
>>
>> * SOLR-11231: Guard against unset fields when performing language
>> detection.
>> @@ -924,6 +925,18 @@ Bug Fixes
>> admin UI dashboard would not display any commandline arguments.
>> (Webster Homer via Shawn Heisey)
>>
>> +* SOLR-11608: Correctly parse the new core-name in the V2 core rename
>> API.
>> + (Jason Gerlowski via Anshum Gupta)
>> +
>> +* SOLR-11256: The queue size for ConcurrentUpdateSolrClient should
>> default to 10 instead of throwing an
>> + IllegalArgumentException. (Jason Gerlowski, Anshum Gupta)
>> +
>> +* SOLR-11616: Snapshot the segments more robustly such that segments
>> created during a backup does does not fail the
>> + operation (Varun Thacker)
>> +
>> +* SOLR-11687: SolrCore.getNewIndexDir falsely returns {dataDir}/index on
>> any IOException reading index.properties
>> + (Nikolay Martynov, Erick Erickson)
>> +
>> * SOLR-9137: bin/solr script ignored custom STOP_PORT on shutdown.
>> (Joachim Kohlhammer, Steve Rowe, Christine Poerschke)
>>
>> @@ -938,18 +951,6 @@ Bug Fixes
>> * SOLR-11691: V2 requests for create-alias didn't work when the
>> collections param was an array.
>> (Jason Gerlowski, Gus Heck, David Smiley, noble)
>>
>> -* SOLR-11608: Correctly parse the new core-name in the V2 core rename
>> API.
>> - (Jason Gerlowski via Anshum Gupta)
>> -
>> -* SOLR-11256: The queue size for ConcurrentUpdateSolrClient should
>> default to 10 instead of throwing an
>> - IllegalArgumentException. (Jason Gerlowski, Anshum Gupta)
>> -
>> -* SOLR-11616: Snapshot the segments more robustly such that segments
>> created during a backup does does not fail the
>> - operation (Varun Thacker)
>> -
>> -* SOLR-11687: SolrCore.getNewIndexDir falsely returns {dataDir}/index on
>> any IOException reading index.properties
>> - (Nikolay Martynov, Erick Erickson)
>> -
>> Optimizations
>> ----------------------
>> * SOLR-11285: Refactor autoscaling framework to avoid direct references
>> to Zookeeper and Solr
>> @@ -1965,12 +1966,12 @@ Other Changes
>> * SOLR-11068: MOVEREPLICA and REPLACENODE API parameter names are now
>> 'sourceNode' and 'targetNode'. The old names
>> viz. 'fromNode' for MOVEREPLICA and 'source', 'target' for REPLACENODE
>> have been deprecated. (shalin)
>>
>> +* SOLR-11088: Fix sporadic failures of
>> MetricsHandlerTest.testPropertyFilter on jenkins (shalin)
>> +
>> * SOLR-11037: Refactor to provide NodeConfig.getSolrDataHome internal
>> API. (ab, janhoy, shalin)
>>
>> * SOLR-11119: Switch from Trie to Points field types in the .system
>> collection schema. (Steve Rowe)
>>
>> -* SOLR-11088: Fix sporadic failures of
>> MetricsHandlerTest.testPropertyFilter on jenkins (shalin)
>> -
>> * SOLR-10494: Make default response format JSON (wt=json), and also
>> indent text responses formats
>> (indent=on) by default (Trey Grainger & Cassandra Targett via hossman)
>>
>> @@ -2070,6 +2071,25 @@ Other Changes
>>
>> * SOLR-11324: Clean up mention of trie fields in documentation and
>> source comments. (Steve Rowe)
>>
>> +================== 6.6.4 ==================
>> +
>> +Consult the LUCENE_CHANGES.txt file for additional, low level, changes
>> in this release.
>> +
>> +Versions of Major Components
>> +---------------------
>> +Apache Tika 1.17
>> +Carrot2 3.15.0
>> +Velocity 1.7 and Velocity Tools 2.0
>> +Apache UIMA 2.3.1
>> +Apache ZooKeeper 3.4.11
>> +Jetty 9.4.10.v20180503
>> +
>> +Bug Fixes
>> +----------------------
>> +
>> +* SOLR-12316: Do not allow to use absolute URIs for including other
>> files in solrconfig.xml and schema parsing.
>> + (Ananthesh, Ishan Chattopadhyaya, Uwe Schindler)
>> +
>> ================== 6.6.3 ==================
>>
>> Consult the LUCENE_CHANGES.txt file for additional, low level, changes
>> in this release.
>>
>>
>> http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a875300a/solr/core/src/test/org/apache/solr/cloud/autoscaling/AutoScalingHandlerTest.java
>> ----------------------------------------------------------------------
>> diff --git
>> a/solr/core/src/test/org/apache/solr/cloud/autoscaling/AutoScalingHandlerTest.java
>> b/solr/core/src/test/org/apache/solr/cloud/autoscaling/AutoScalingHandlerTest.java
>> index 2aec88e..483b60c 100644
>> ---
>> a/solr/core/src/test/org/apache/solr/cloud/autoscaling/AutoScalingHandlerTest.java
>> +++
>> b/solr/core/src/test/org/apache/solr/cloud/autoscaling/AutoScalingHandlerTest.java
>> @@ -757,14 +757,14 @@ public class AutoScalingHandlerTest extends
>> SolrCloudTestCase {
>> response = solrClient.request(req);
>>
>> Map<String, Object> diagnostics = (Map<String, Object>)
>> response.get("diagnostics");
>> - List sortedNodes = (List) diagnostics.get("sortedNodes");
>> + List sortedNodes = (List) Utils.getObjectByPath(response, false,
>> "diagnostics/sortedNodes");
>> assertNotNull(sortedNodes);
>>
>> assertEquals(2, sortedNodes.size());
>> for (int i = 0; i < 2; i++) {
>> Map node = (Map) sortedNodes.get(i);
>> assertNotNull(node);
>> - assertEquals(5, node.size());
>> + assertEquals(6, node.size());
>> assertNotNull(node.get("node"));
>> assertNotNull(node.get("cores"));
>> assertEquals(0L, node.get("cores"));
>>
>>
>> http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a875300a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
>> ----------------------------------------------------------------------
>> diff --git
>> a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
>> b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
>> index 05c9c20..fb01cc5 100644
>> ---
>> a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
>> +++
>> b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Policy.java
>> @@ -25,6 +25,7 @@ import java.util.Collection;
>> import java.util.Collections;
>> import java.util.HashMap;
>> import java.util.HashSet;
>> +import java.util.Iterator;
>> import java.util.LinkedHashMap;
>> import java.util.List;
>> import java.util.Map;
>> @@ -359,14 +360,35 @@ public class Policy implements MapWriter {
>> }
>>
>> static void setApproxValuesAndSortNodes(List<Preference>
>> clusterPreferences, List<Row> matrix) {
>> + List<Row> deadNodes = null;
>> + Iterator<Row> it =matrix.iterator();
>> + while (it.hasNext()){
>> + Row row = it.next();
>> + if(!row.isLive){
>> + if(deadNodes == null) deadNodes = new ArrayList<>();
>> + deadNodes.add(row);
>> + it.remove();
>> + }
>> + }
>> +
>> if (!clusterPreferences.isEmpty()) {
>> //this is to set the approximate value according to the precision
>> ArrayList<Row> tmpMatrix = new ArrayList<>(matrix);
>> + Row[] lastComparison = new Row[2];
>> for (Preference p : clusterPreferences) {
>> try {
>> - tmpMatrix.sort((r1, r2) -> p.compare(r1, r2, false));
>> + tmpMatrix.sort((r1, r2) -> {
>> + lastComparison[0] = r1;
>> + lastComparison[1] = r2;
>> + return p.compare(r1, r2, false);
>> + });
>> } catch (Exception e) {
>> - LOG.error("Exception! prefs = {}, matrix = {}",
>> clusterPreferences, matrix);
>> + LOG.error("Exception! prefs = {}, recent r1 = {}, r2 = {},
>> compare : {} matrix = {}",
>> + clusterPreferences,
>> + lastComparison[0].node,
>> + lastComparison[1].node,
>> + p.compare(lastComparison[0],lastComparison[1], false ),
>> + Utils.toJSONString(Utils.getDeepCopy(tmpMatrix, 6,
>> false)));
>> throw e;
>> }
>> p.setApproxVal(tmpMatrix);
>> @@ -378,6 +400,12 @@ public class Policy implements MapWriter {
>> if (result == 0) result = clusterPreferences.get(0).compare(r1,
>> r2, false);
>> return result;
>> });
>> +
>> + if(deadNodes != null){
>> + for (Row deadNode : deadNodes) {
>> + matrix.add(0, deadNode);
>> + }
>> + }
>> }
>> }
>>
>>
>>
>> http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a875300a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/PolicyHelper.java
>> ----------------------------------------------------------------------
>> diff --git
>> a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/PolicyHelper.java
>> b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/PolicyHelper.java
>> index 11c43d5..112ae76 100644
>> ---
>> a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/PolicyHelper.java
>> +++
>> b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/PolicyHelper.java
>> @@ -190,6 +190,7 @@ public class PolicyHelper {
>> List<Map<String, Object>> sortedNodes = new
>> ArrayList<>(sorted.size());
>> for (Row row : sorted) {
>> Map<String, Object> map = Utils.makeMap("node", row.node);
>> + map.put("isLive", row.isLive);
>> for (Cell cell : row.getCells()) {
>> for (Preference clusterPreference : clusterPreferences) {
>> Policy.SortParam name = clusterPreference.getName();
>>
>>
>> http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a875300a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Preference.java
>> ----------------------------------------------------------------------
>> diff --git
>> a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Preference.java
>> b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Preference.java
>> index 3e73632..0db9886 100644
>> ---
>> a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Preference.java
>> +++
>> b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Preference.java
>> @@ -63,9 +63,6 @@ public class Preference implements MapWriter {
>> // recursive, it uses the precision to tie & when there is a tie use
>> the next preference to compare
>> // in non-recursive mode, precision is not taken into consideration
>> and sort is done on actual value
>> int compare(Row r1, Row r2, boolean useApprox) {
>> - if (!r1.isLive && !r2.isLive) return 0;
>> - if (!r1.isLive) return -1;
>> - if (!r2.isLive) return 1;
>> Object o1 = useApprox ? r1.cells[idx].approxVal : r1.cells[idx].val;
>> Object o2 = useApprox ? r2.cells[idx].approxVal : r2.cells[idx].val;
>> int result = 0;
>>
>>
>> http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a875300a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Row.java
>> ----------------------------------------------------------------------
>> diff --git
>> a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Row.java
>> b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Row.java
>> index 659f315..f1799c4 100644
>> ---
>> a/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Row.java
>> +++
>> b/solr/solrj/src/java/org/apache/solr/client/solrj/cloud/autoscaling/Row.java
>> @@ -19,6 +19,7 @@ package org.apache.solr.client.solrj.cloud.autoscaling;
>>
>> import java.io.IOException;
>> import java.util.ArrayList;
>> +import java.util.Arrays;
>> import java.util.Collections;
>> import java.util.HashMap;
>> import java.util.List;
>> @@ -27,7 +28,6 @@ import java.util.Random;
>> import java.util.function.Consumer;
>> import java.util.stream.Collectors;
>>
>> -import org.apache.solr.common.IteratorWriter;
>> import org.apache.solr.common.MapWriter;
>> import org.apache.solr.common.cloud.Replica;
>> import org.apache.solr.common.cloud.ZkStateReader;
>> @@ -81,10 +81,10 @@ public class Row implements MapWriter {
>>
>> @Override
>> public void writeMap(EntryWriter ew) throws IOException {
>> - ew.put(node, (IteratorWriter) iw -> {
>> - iw.add((MapWriter) e -> e.put("replicas",
>> collectionVsShardVsReplicas));
>> - for (Cell cell : cells) iw.add(cell);
>> - });
>> + ew.put(NODE, node);
>> + ew.put("replicas", collectionVsShardVsReplicas);
>> + ew.put("isLive", isLive);
>> + ew.put("attributes", Arrays.asList(cells));
>> }
>>
>> Row copy(Policy.Session session) {
>>
>>
>> http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a875300a/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
>> ----------------------------------------------------------------------
>> diff --git
>> a/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
>> b/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
>> index a53b60c..a099171 100644
>> ---
>> a/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
>> +++
>> b/solr/solrj/src/test/org/apache/solr/client/solrj/cloud/autoscaling/TestPolicy.java
>> @@ -2160,5 +2160,313 @@ public void testUtilizeNodeFailure2() throws
>> Exception {
>> assertEquals("count = "+count ,1,count);
>> }
>>
>> -
>> + //SOLR-12358
>> + public void testSortError() {
>> + Policy policy = new Policy((Map<String, Object>)
>> Utils.fromJSONString("{cluster-preferences: [{minimize : cores,
>> precision:1}, " +
>> + "{maximize : freedisk, precision: 50}, " +
>> + "{minimize: sysLoadAvg}]}"));
>> + String rowsData = "{'sortedNodes':[" +
>> + " {" +
>> + " 'node':'solr-01:8983_solr'," +
>> + " 'replicas':{}," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':2}," +
>> + " {'freedisk':1734.5261459350586}," +
>> + " {'sysLoadAvg':35.0}," +
>> + " {'node':'solr-01:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-07:8983_solr'," +
>> + " 'replicas':{}," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1721.5669250488281}," +
>> + " {'sysLoadAvg':10.0}," +
>> + " {'node':'solr-07:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-08:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1764.9518203735352}," +
>> + " {'sysLoadAvg':330.0}," +
>> + " {'node':'solr-08:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-25:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1779.7792778015137}," +
>> + " {'sysLoadAvg':304.0}," +
>> + " {'node':'solr-25:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-15:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1697.5930519104004}," +
>> + " {'sysLoadAvg':277.0}," +
>> + " {'node':'solr-15:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-13:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':2}," +
>> + " {'freedisk':1755.1909484863281}," +
>> + " {'sysLoadAvg':265.0}," +
>> + " {'node':'solr-13:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-14:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1757.6035423278809}," +
>> + " {'sysLoadAvg':61.0}," +
>> + " {'node':'solr-14:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-16:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1746.081386566162}," +
>> + " {'sysLoadAvg':260.0}," +
>> + " {'node':'solr-16:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-04:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':2}," +
>> + " {'freedisk':1708.7230529785156}," +
>> + " {'sysLoadAvg':216.0}," +
>> + " {'node':'solr-04:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-06:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1688.3182678222656}," +
>> + " {'sysLoadAvg':385.0}," +
>> + " {'node':'solr-06:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-02:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':6}," +
>> + " {'freedisk':1778.226963043213}," +
>> + " {'sysLoadAvg':369.0}," +
>> + " {'node':'solr-02:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-05:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1741.9401931762695}," +
>> + " {'sysLoadAvg':354.0}," +
>> + " {'node':'solr-05:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-23:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1718.854579925537}," +
>> + " {'sysLoadAvg':329.0}," +
>> + " {'node':'solr-23:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-24:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1733.6669311523438}," +
>> + " {'sysLoadAvg':327.0}," +
>> + " {'node':'solr-24:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-09:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1714.6191711425781}," +
>> + " {'sysLoadAvg':278.0}," +
>> + " {'node':'solr-09:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-10:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1755.3038482666016}," +
>> + " {'sysLoadAvg':266.0}," +
>> + " {'node':'solr-10:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-28:8983_solr'," +
>> + " 'isLive':false," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1691.3830909729004}," +
>> + " {'sysLoadAvg':261.0}," +
>> + " {'node':'solr-28:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-29:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':2}," +
>> + " {'freedisk':1706.797966003418}," +
>> + " {'sysLoadAvg':252.99999999999997}," +
>> + " {'node':'solr-29:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-32:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1762.432300567627}," +
>> + " {'sysLoadAvg':221.0}," +
>> + " {'node':'solr-32:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-21:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1760.9801979064941}," +
>> + " {'sysLoadAvg':213.0}," +
>> + " {'node':'solr-21:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-22:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1780.5297241210938}," +
>> + " {'sysLoadAvg':209.0}," +
>> + " {'node':'solr-22:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-31:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1700.1481628417969}," +
>> + " {'sysLoadAvg':211.0}," +
>> + " {'node':'solr-31:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-33:8983_solr'," +
>> + " 'isLive':false," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1748.1132926940918}," +
>> + " {'sysLoadAvg':199.0}," +
>> + " {'node':'solr-33:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-36:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1776.197639465332}," +
>> + " {'sysLoadAvg':193.0}," +
>> + " {'node':'solr-36:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-35:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1746.7729606628418}," +
>> + " {'sysLoadAvg':191.0}," +
>> + " {'node':'solr-35:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-12:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1713.287540435791}," +
>> + " {'sysLoadAvg':175.0}," +
>> + " {'node':'solr-12:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-11:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1736.784511566162}," +
>> + " {'sysLoadAvg':169.0}," +
>> + " {'node':'solr-11:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-35:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1766.9416885375977}," +
>> + " {'sysLoadAvg':155.0}," +
>> + " {'node':'solr-35:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-17:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1764.3425407409668}," +
>> + " {'sysLoadAvg':139.0}," +
>> + " {'node':'solr-17:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-18:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':2}," +
>> + " {'freedisk':1757.0613975524902}," +
>> + " {'sysLoadAvg':132.0}," +
>> + " {'node':'solr-18:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-20:8983_solr'," +
>> + " 'isLive':false," +
>> + " 'attributes':[" +
>> + " {'cores':3}," +
>> + " {'freedisk':1747.4205322265625}," +
>> + " {'sysLoadAvg':126.0}," +
>> + " {'node':'solr-20:8983_solr'}]}," +
>> + " {" +
>> + " 'node':'solr-27:8983_solr'," +
>> + " 'isLive':true," +
>> + " 'attributes':[" +
>> + " {'cores':4}," +
>> + " {'freedisk':1721.0442085266113}," +
>> + " {'sysLoadAvg':118.0}," +
>> + " {'node':'solr-27:8983_solr'}]}]}";
>> +
>> + List l = (List) ((Map)
>> Utils.fromJSONString(rowsData)).get("sortedNodes");
>> + List<Suggestion.ConditionType> params = new ArrayList<>();
>> + params.add(Suggestion.ConditionType.CORES);
>> + params.add(Suggestion.ConditionType.FREEDISK);
>> + params.add(Suggestion.ConditionType.SYSLOADAVG);
>> + params.add(Suggestion.ConditionType.NODE);
>> + List<Row> rows = new ArrayList<>();
>> + for (Object o : l) {
>> + Map m = (Map) o;
>> + Cell[] c = new Cell[params.size()];
>> + List attrs = (List) m.get("attributes");
>> + for (int i = 0; i < params.size(); i++) {
>> + Suggestion.ConditionType param = params.get(i);
>> + for (Object attr : attrs) {
>> + Object o1 = ((Map) attr).get(param.tagName);
>> + if (o1 != null) {
>> + o1 = param.validate(param.tagName, o1, false);
>> + c[i] = new Cell(i, param.tagName, o1, o1, param, null);
>> + }
>> + }
>> + }
>> + rows.add(new Row((String) m.get("node"), c, false,
>> + new HashMap<>(),
>> + (Boolean) m.get("isLive"), null));
>> + }
>> + int deadNodes = 0;
>> + for (Row row : rows) {
>> + if (!row.isLive) deadNodes++;
>> + }
>> +
>> + Policy.setApproxValuesAndSortNodes(policy.clusterPreferences, rows);
>> +
>> + for (int i = 0; i < deadNodes; i++) {
>> + assertFalse(rows.get(i).isLive);
>> + }
>> +
>> + for (int i = deadNodes; i < rows.size(); i++) {
>> + assertTrue(rows.get(i).isLive);
>> + }
>> +
>> +
>> + }
>> +
>> +
>> }
>>
>>