You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2020/09/14 12:08:21 UTC
[GitHub] [lucene-solr] sigram commented on a change in pull request #1845: SOLR-14613: Autoscaling replacement using placement plugins
sigram commented on a change in pull request #1845:
URL: https://github.com/apache/lucene-solr/pull/1845#discussion_r486531558
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/Cluster.java
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.cluster.placement;
+
+import java.io.IOException;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * <p>A representation of the (initial) cluster state, providing information on which nodes are part of the cluster and a way
+ * to get to more detailed info.
+ */
+public interface Cluster {
+ /**
+ * @return current set of live nodes. Never <code>null</code>, never empty (Solr wouldn't call the plugin if empty
+ * since no useful work could then be done).
+ */
+ Set<Node> getLiveNodes();
+
+ /**
+ * <p>Returns info about the given collection if one exists. Because it is not expected for plugins to request info about
+ * a large number of collections, requests can only be made one by one.
+ *
+ * <p>This is also the reason we do not return a {@link java.util.Map} or {@link Set} of {@link SolrCollection}'s here: it would be
+ * wasteful to fetch all data and fill such a map when plugin code likely needs info about at most one or two collections.
+ */
+ Optional<SolrCollection> getCollection(String collectionName) throws IOException;
+
+ /**
+ * <p>Allows getting names of all {@link SolrCollection}'s present in the cluster.
+ *
+ * <p><b>WARNING:</b> this call will be extremely inefficient on large clusters. Usage is discouraged.
+ */
+ Set<String> getAllCollectionNames();
Review comment:
IIRC at some point we've considered using an Iterator<String> here instead.
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/Node.java
##########
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.cluster.placement;
+
+/**
+ * Representation of a SolrCloud node or server in the SolrCloud cluster.
+ */
+public interface Node {
Review comment:
So ... given that there's already a `SolrNode` interface in master, which already provides isolation from implementation details, shouldn't we use that here? The same applies to `SolrCollection` and `ShardReplica`.
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/SolrCollection.java
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.cluster.placement;
+
+import java.util.Iterator;
+
+/**
+ * Represents a Collection in SolrCloud (unrelated to {@link java.util.Collection} that uses the nicer name).
+ */
+public interface SolrCollection {
Review comment:
See my other comments about merging this with the existing `SolrCollection`.
##########
File path: solr/solrj/src/resources/apispec/cluster.Commands.json
##########
@@ -141,6 +141,21 @@
}
}
}
- }
+ },
+ "set-placement-plugin": {
Review comment:
@noblepaul do we still need these awful json apispecs if we use the V2 API annotations?
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/Replica.java
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.cluster.placement;
+
+/**
+ * An instantiation (or one of the copies) of a given {@link Shard} of a given {@link SolrCollection}.
+ * Objects of this type are returned by the Solr framework to the plugin, they are not directly built by the plugin. When the
+ * plugin wants to add a replica it goes through appropriate method in {@link PlacementPlanFactory}).
+ */
+public interface Replica {
Review comment:
This should be merged with the existing `ShardReplica` to avoid creating separate abstractions for each subsystem.
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/ReplicaPlacement.java
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.cluster.placement;
+
+/**
+ * <p>Placement decision for a single {@link Replica}. Note this placement decision is used as part of a {@link PlacementPlan},
+ * it does not directly lead to the plugin code getting a corresponding {@link Replica} instance, nor does it require the
+ * plugin to provide a {@link Shard} instance (the plugin code gets such instances for existing replicas and shards in the
+ * cluster but does not create them directly for adding new replicas for new or existing shards).
+ *
+ * <p>Captures the {@link Shard} (via the shard name), {@link Node} and {@link Replica.ReplicaType} of a Replica to be created.
+ *
+ * <p>TODO: discuss (before merge) if this interface really needs to allow access to the data captured in an instance or if calling {@link PlacementPlanFactory#createReplicaPlacement(String, Node, Replica.ReplicaType)} is sufficient for plugin code.
+ */
+public interface ReplicaPlacement {
+ /**
+ * @return the name of the {@link Shard} for which the replica should be created
+ */
+ String getShardName();
Review comment:
Should we also have the collection name here for completeness?
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/PlacementRequest.java
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.cluster.placement;
+
+import java.util.Set;
+
+/**
+ * A cluster related placement request that Solr asks a {@link PlacementPlugin} plugin to resolve and compute a
+ * {@link PlacementPlan} placing one or more {@link Replica}'s of one or more {@link Shard}'s of an existing {@link SolrCollection}.
+ * The shard might or might not already exist, plugin code can easily find out by calling {@link SolrCollection#getShard(String)}
+ * with the shard name(s) returned by {@link #getShardNames()}.
+ *
+ * <p>The set of {@link Node}s on which the replicas should be placed
+ * is specified (defaults to being equal to the set returned by {@link Cluster#getLiveNodes()}).
+ */
+public interface PlacementRequest {
+ /**
+ * The {@link SolrCollection} to add {@link Replica}(s) to.
+ */
+ SolrCollection getCollection();
+
+ /**
+ * <p>Shard name(s) for which new replicas placement should be computed. The shard(s) might exist or not (that's why this
+ * method returns a {@link Set} of {@link String}'s and not directly a set of {@link Shard} instances).
+ *
+ * <p>Note the Collection API allows specifying the shard name or a {@code _route_} parameter. The Solr implementation will
+ * convert either specification into the relevant shard name so the plugin code doesn't have to worry about this.
+ */
+ Set<String> getShardNames();
+
+ /**
+ * <p>Replicas should only be placed on nodes in the set returned by this method.
+ *
+ * <p>When Collection API calls do not specify a specific set of target nodes, replicas can be placed on any live node of
+ * the cluster. In such cases, this set will be equal to the set of all live nodes. The plugin placement code does not
+ * need to worry (or care) if a set of nodes was explicitly specified or not.
+ *
+ * @return never {@code null} and never empty set (if that set was to be empty for any reason, no placement would be
+ * possible and the Solr infrastructure driving the plugin code would detect the error itself rather than calling the plugin).
+ */
+ Set<Node> getTargetNodes();
+
+ /**
+ * Returns the number of replica to create that is returned by the corresponding method {@link #getCountNrtReplicas()},
+ * {@link #getCountTlogReplicas()} or {@link #getCountPullReplicas()}. Might delete the other three.
+ */
+ int getCountReplicasToCreate(Replica.ReplicaType replicaType);
Review comment:
I slightly prefer this method, as it allows us to modify available replica types without changing the interface.
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/Shard.java
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.cluster.placement;
+
+import java.util.Iterator;
+
+/**
+ * Shard in a {@link SolrCollection}, i.e. a subset of the data indexed in that collection.
+ */
+public interface Shard {
Review comment:
Similar to the other top-level abstractions, this interface should be merged with the existing `Shard` interface, after resolving the main differences (the use of iterators vs. `SimpleMap`, what getters we absolutely need in this interface, etc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org