You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by GitBox <gi...@apache.org> on 2021/04/06 15:50:59 UTC
[GitHub] [solr] sigram opened a new pull request #60: SOLR-15232: Add replica(s) as a part of node startup
sigram opened a new pull request #60:
URL: https://github.com/apache/solr/pull/60
The solution implemented in this PR is a loose port of the `SimScenario` package from Solr 8.x, which was designed for running reproducible scenarios for the autoscaling framework.
I re-worked the DSL and removed commands that are not applicable in the current context. The functionality is as follows:
* two scripts can be supplied using system properties: `-DinitScript=...` and `-DshutdownScript=...`, which specify a text file resource (either absolute or relative to Solr home).
* each script consists of series of operations that perform collection commands (plus some rudimentary conditionals, wait for state, asserts, etc)
* operation parameters may use variable substitution, using either system properties or references to results of the previous operations. A shared context is provided where scripts may keep additional state between operations.
* several context variables are filled in based on the current node environment.
Example of an init script that uses a context variable `_node_name_` and a set of system properties defined by user: `collection`, `shard` to add a replica after startup and make sure it's operational - and if not the script will shut down the `CoreContainer`:
```
# any errors will result in CC shutdown
set key=_error_handling_&value=FATAL
# request ADDREPLICA using sysprop values for collection and shard, and a context var for this node name
request /admin/collections?action=ADDREPLICA&collection=${collection}&shard=${shard}&type=NRT&node=${_node_name_}
# put the resulting core name in a context var 'newReplica'
set key=newReplica&value=${_last_result_/result/success[0]/value/core}
# wait for the new replica to become active
wait_replica collection=${collection}&core=${newReplica}&state=ACTIVE
# log replica name
log key=_last_result_/result[0]/key&format=******%20Replica%20name:%20{}%20******
# end of script
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] sigram closed pull request #60: SOLR-15232: Add replica(s) as a part of node startup
Posted by GitBox <gi...@apache.org>.
sigram closed pull request #60:
URL: https://github.com/apache/solr/pull/60
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] sigram commented on pull request #60: SOLR-15232: Add replica(s) as a part of node startup
Posted by GitBox <gi...@apache.org>.
sigram commented on pull request #60:
URL: https://github.com/apache/solr/pull/60#issuecomment-839968476
In light of the comments I'm closing this as Won't Do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] murblanc commented on a change in pull request #60: SOLR-15232: Add replica(s) as a part of node startup
Posted by GitBox <gi...@apache.org>.
murblanc commented on a change in pull request #60:
URL: https://github.com/apache/solr/pull/60#discussion_r608666371
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -1031,6 +1070,9 @@ public void shutdown() {
if (zkController != null) {
OverseerTaskQueue overseerCollectionQueue = zkController.getOverseerCollectionQueue();
overseerCollectionQueue.allowOverseerPendingTasksToComplete();
Review comment:
We're first letting the collection queue drain then add new collection api items?
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -930,10 +931,48 @@ public void load() {
clusterSingletons.setReady();
zkSys.getZkController().checkOverseerDesignate();
-
}
// This is a bit redundant but these are two distinct concepts for all they're accomplished at the same time.
status |= LOAD_COMPLETE | INITIAL_CORE_LOAD_COMPLETE;
+
+ if (isZooKeeperAware()) {
+ // run init script if needed
+ String resourceName = System.getProperty(Script.INIT_SCRIPT);
+ runScript(resourceName, false);
+ }
+ }
+
+ public void runScript(String resourceName, boolean inShutdown) {
Review comment:
This doesn't seem like it belongs in CoreContainer. Maybe CoreContainer is passed an interface that is called upon startup/shutdown and that hides what's actually done, (if anything is done) so the script machinery is completely independent of CoreContainer.
The callbacks should likely be called whether in Cloud mode or not, and let the actual callback implementation decide if something should be done or not... There are likely uses for such execution hooks in non Cloud mode as well...
##########
File path: solr/core/src/java/org/apache/solr/util/scripting/Script.java
##########
@@ -0,0 +1,1011 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.util.scripting;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.cloud.SolrCloudManager;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.impl.ClusterStateProvider;
+import org.apache.solr.client.solrj.impl.SolrClientCloudManager;
+import org.apache.solr.client.solrj.request.GenericSolrRequest;
+import org.apache.solr.client.solrj.request.RequestWriter;
+import org.apache.solr.cloud.CloudUtil;
+import org.apache.solr.common.MapWriter;
+import org.apache.solr.common.cloud.ClusterState;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.ExecutorUtil;
+import org.apache.solr.common.util.PropertiesUtil;
+import org.apache.solr.common.util.SolrNamedThreadFactory;
+import org.apache.solr.common.util.TimeSource;
+import org.apache.solr.common.util.Utils;
+import org.apache.solr.core.SolrResourceLoader;
+import org.apache.solr.util.TimeOut;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.invoke.MethodHandles;
+import java.net.URLDecoder;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Random;
+import java.util.Stack;
+import java.util.TreeMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Function;
+
+/**
+ * This class represents a script consisting of a series of operations.
+ */
+public class Script implements Closeable {
Review comment:
This class definitely does not feel like it belongs in Solr core. The whole concept being implemented here feels more like a high level "trigger" for a cluster (or a node? Running the script on each node does feel brittle, esp. in setups where the number of nodes is variable with elasticity) than something that should be integrated into the node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org