You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by il...@apache.org on 2020/11/20 19:45:03 UTC
[lucene-solr] branch jira/solr-15004 updated: Move plugin Javadoc
from private plugin to public factory
This is an automated email from the ASF dual-hosted git repository.
ilan pushed a commit to branch jira/solr-15004
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
The following commit(s) were added to refs/heads/jira/solr-15004 by this push:
new 0cfcf83 Move plugin Javadoc from private plugin to public factory
0cfcf83 is described below
commit 0cfcf837c4c9b881d671da65b0584358280c34c4
Author: Ilan Ginzburg <ig...@salesforce.com>
AuthorDate: Fri Nov 20 20:44:15 2020 +0100
Move plugin Javadoc from private plugin to public factory
---
.../plugins/AffinityPlacementFactory.java | 94 +++++++++++-----------
.../plugins/MinimizeCoresPlacementFactory.java | 8 +-
.../placement/plugins/RandomPlacementFactory.java | 6 +-
.../impl/AffinityPlacementFactoryTest.java | 2 +-
4 files changed, 54 insertions(+), 56 deletions(-)
diff --git a/solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java b/solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
index db968e8..50ecb8d 100644
--- a/solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
+++ b/solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
@@ -67,6 +67,52 @@ import java.util.stream.Collectors;
"set-placement-plugin" : null
}' http://localhost:8983/api/cluster
* </pre>
+ *
+ *
+ * <p>{@link AffinityPlacementPlugin} implements placing replicas in a way that replicate past Autoscaling config defined
+ * <a href="https://github.com/lucidworks/fusion-cloud-native/blob/master/policy.json#L16">here</a>.</p>
+ *
+ * <p>This specification is doing the following:
+ * <p><i>Spread replicas per shard as evenly as possible across multiple availability zones (given by a sys prop),
+ * assign replicas based on replica type to specific kinds of nodes (another sys prop), and avoid having more than
+ * one replica per shard on the same node.<br>
+ * Only after these constraints are satisfied do minimize cores per node or disk usage.</i></p>
+ *
+ * <p>Overall strategy of this plugin:</p>
+ * <ul><li>
+ * The set of nodes in the cluster is obtained and transformed into 3 independent sets (that can overlap) of nodes
+ * accepting each of the three replica types.
+ * </li><li>
+ * For each shard on which placing replicas is required and then for each replica type to place (starting with NRT,
+ * then TLOG then PULL): <ul>
+ * <li>The set of candidates nodes corresponding to the replica type is used and from that set are removed nodes
+ * that already have a replica (of any type) for that shard</li>
+ * <li>If there are not enough nodes, an error is thrown (this is checked further down during processing).</li>
+ * <li>The number of (already existing) replicas of the current type on each Availability Zone is collected.</li>
+ * <li>Separate the set of available nodes to as many subsets (possibly some are empty) as there are Availability Zones
+ * defined for the candidate nodes</li>
+ * <li>In each AZ nodes subset, sort the nodes by increasing total number of cores count, with possibly a condition
+ * that pushes nodes with low disk space to the end of the list? Or a weighted combination of the relative
+ * importance of these two factors? Some randomization? Marking as non available nodes with not enough disk space?
+ * These and other are likely aspects to be played with once the plugin is tested or observed to be running in prod,
+ * don't expect the initial code drop(s) to do all of that.</li>
+ * <li>Iterate over the number of replicas to place (for the current replica type for the current shard):
+ * <ul>
+ * <li>Based on the number of replicas per AZ collected previously, pick the non empty set of nodes having the
+ * lowest number of replicas. Then pick the first node in that set. That's the node the replica is placed one.
+ * Remove the node from the set of available nodes for the given AZ and increase the number of replicas placed
+ * on that AZ.</li>
+ * </ul></li>
+ * <li>During this process, the number of cores on the nodes in general is tracked to take into account placement
+ * decisions so that not all shards decide to put their replicas on the same nodes (they might though if these are
+ * the less loaded nodes).</li>
+ * </ul>
+ * </li>
+ * </ul>
+ *
+ * <p>This code is a realistic placement computation, based on a few assumptions. The code is written in such a way to
+ * make it relatively easy to adapt it to (somewhat) different assumptions. Configuration options could be introduced
+ * to allow configuration base option selection as well...</p>
*/
public class AffinityPlacementFactory implements PlacementPluginFactory {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
@@ -88,52 +134,8 @@ public class AffinityPlacementFactory implements PlacementPluginFactory {
}
/**
- * <p>Implements placing replicas in a way that replicate past Autoscaling config defined
- * <a href="https://github.com/lucidworks/fusion-cloud-native/blob/master/policy.json#L16">here</a>.</p>
- *
- * <p>This specification is doing the following:
- * <p><i>Spread replicas per shard as evenly as possible across multiple availability zones (given by a sys prop),
- * assign replicas based on replica type to specific kinds of nodes (another sys prop), and avoid having more than
- * one replica per shard on the same node.<br>
- * Only after these constraints are satisfied do minimize cores per node or disk usage.</i></p>
- *
- * <p>Overall strategy of this plugin:</p>
- * <ul><li>
- * The set of nodes in the cluster is obtained and transformed into 3 independent sets (that can overlap) of nodes
- * accepting each of the three replica types.
- * </li><li>
- * For each shard on which placing replicas is required and then for each replica type to place (starting with NRT,
- * then TLOG then PULL): <ul>
- * <li>The set of candidates nodes corresponding to the replica type is used and from that set are removed nodes
- * that already have a replica (of any type) for that shard</li>
- * <li>If there are not enough nodes, an error is thrown (this is checked further down during processing).</li>
- * <li>The number of (already existing) replicas of the current type on each Availability Zone is collected.</li>
- * <li>Separate the set of available nodes to as many subsets (possibly some are empty) as there are Availability Zones
- * defined for the candidate nodes</li>
- * <li>In each AZ nodes subset, sort the nodes by increasing total number of cores count, with possibly a condition
- * that pushes nodes with low disk space to the end of the list? Or a weighted combination of the relative
- * importance of these two factors? Some randomization? Marking as non available nodes with not enough disk space?
- * These and other are likely aspects to be played with once the plugin is tested or observed to be running in prod,
- * don't expect the initial code drop(s) to do all of that.</li>
- * <li>Iterate over the number of replicas to place (for the current replica type for the current shard):
- * <ul>
- * <li>Based on the number of replicas per AZ collected previously, pick the non empty set of nodes having the
- * lowest number of replicas. Then pick the first node in that set. That's the node the replica is placed one.
- * Remove the node from the set of available nodes for the given AZ and increase the number of replicas placed
- * on that AZ.</li>
- * </ul></li>
- * <li>During this process, the number of cores on the nodes in general is tracked to take into account placement
- * decisions so that not all shards decide to put their replicas on the same nodes (they might though if these are
- * the less loaded nodes).</li>
- * </ul>
- * </li>
- * </ul>
- *
- * <p>This code is a realistic placement computation, based on a few assumptions. The code is written in such a way to
- * make it relatively easy to adapt it to (somewhat) different assumptions. Configuration options could be introduced
- * to allow configuration base option selection as well...</p>
- *
- * See {@link AffinityPlacementFactory} for instructions on how to configure a cluster to use this plugin.
+ * See {@link AffinityPlacementFactory} for instructions on how to configure a cluster to use this plugin and details
+ * on what the plugin does.
*/
static private class AffinityPlacementPlugin implements PlacementPlugin {
/**
diff --git a/solr/core/src/java/org/apache/solr/cluster/placement/plugins/MinimizeCoresPlacementFactory.java b/solr/core/src/java/org/apache/solr/cluster/placement/plugins/MinimizeCoresPlacementFactory.java
index 33d9c57..b73b692 100644
--- a/solr/core/src/java/org/apache/solr/cluster/placement/plugins/MinimizeCoresPlacementFactory.java
+++ b/solr/core/src/java/org/apache/solr/cluster/placement/plugins/MinimizeCoresPlacementFactory.java
@@ -34,7 +34,9 @@ import org.apache.solr.cluster.placement.*;
import org.apache.solr.common.util.SuppressForbidden;
/**
- * <p>Factory for creating {@link MinimizeCoresPlacementPlugin}.</p>
+ * <p>Factory for creating {@link MinimizeCoresPlacementPlugin}, a Placement plugin implementing placing replicas
+ * to minimize number of cores per {@link Node}, while not placing two replicas of the same shard on the same node.
+ * This code is meant as an educational example of a placement plugin.</p>
*
* <p>See {@link AffinityPlacementFactory} for a more realistic example and documentation.</p>
*/
@@ -45,10 +47,6 @@ public class MinimizeCoresPlacementFactory implements PlacementPluginFactory {
return new MinimizeCoresPlacementPlugin();
}
- /**
- * Placement plugin implementing placing replicas to minimize number of cores per {@link Node}, while not placing two
- * replicas of the same shard on the same node. This code is meant as an educational example of a placement plugin.
- */
static private class MinimizeCoresPlacementPlugin implements PlacementPlugin {
@SuppressForbidden(reason = "Ordering.arbitrary() has no equivalent in Comparator class. Rather reuse than copy.")
diff --git a/solr/core/src/java/org/apache/solr/cluster/placement/plugins/RandomPlacementFactory.java b/solr/core/src/java/org/apache/solr/cluster/placement/plugins/RandomPlacementFactory.java
index 7874c33..b9d3884 100644
--- a/solr/core/src/java/org/apache/solr/cluster/placement/plugins/RandomPlacementFactory.java
+++ b/solr/core/src/java/org/apache/solr/cluster/placement/plugins/RandomPlacementFactory.java
@@ -30,7 +30,8 @@ import org.apache.solr.cluster.SolrCollection;
import org.apache.solr.cluster.placement.*;
/**
- * <p>Factory for creating {@link RandomPlacementPlugin}.</p>
+ * <p>Factory for creating {@link RandomPlacementPlugin}, a placement plugin implementing random placement for new
+ * collection creation while preventing two replicas of same shard from being placed on same node..</p>
*
* <p>See {@link AffinityPlacementFactory} for a more realistic example and documentation.</p>
*/
@@ -41,9 +42,6 @@ public class RandomPlacementFactory implements PlacementPluginFactory {
return new RandomPlacementPlugin();
}
- /**
- * Implements random placement for new collection creation while preventing two replicas of same shard from being placed on same node.
- */
static private class RandomPlacementPlugin implements PlacementPlugin {
public PlacementPlan computePlacement(Cluster cluster, PlacementRequest request, AttributeFetcher attributeFetcher,
diff --git a/solr/core/src/test/org/apache/solr/cluster/placement/impl/AffinityPlacementFactoryTest.java b/solr/core/src/test/org/apache/solr/cluster/placement/impl/AffinityPlacementFactoryTest.java
index 48c7eb6..80f30105 100644
--- a/solr/core/src/test/org/apache/solr/cluster/placement/impl/AffinityPlacementFactoryTest.java
+++ b/solr/core/src/test/org/apache/solr/cluster/placement/impl/AffinityPlacementFactoryTest.java
@@ -165,7 +165,7 @@ public class AffinityPlacementFactoryTest extends Assert {
long start = System.nanoTime();
PlacementPlan pp = plugin.computePlacement(cluster, placementRequest, attributeFetcher, placementPlanFactory);
long end = System.nanoTime();
- log.info("ComputePlacement: {} nodes, {} shards, {} total replicas, elapsed time {} ms.", numNodes, numShards, TOTAL_REPLICAS, TimeUnit.NANOSECONDS.toMillis(end - start));
+ log.info("ComputePlacement: {} nodes, {} shards, {} total replicas, elapsed time {} ms.", numNodes, numShards, TOTAL_REPLICAS, TimeUnit.NANOSECONDS.toMillis(end - start)); //nowarn
assertEquals("incorrect number of calculated placements", TOTAL_REPLICAS,
pp.getReplicaPlacements().size());
// check that replicas are correctly placed