You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by dw...@apache.org on 2021/03/10 09:59:44 UTC
[lucene] 31/50: SOLR-11165: Ref guide documentation for Autoscaling
APIs, triggers, listeners, actions for Solr 7.1
This is an automated email from the ASF dual-hosted git repository.
dweiss pushed a commit to branch branch_7_1
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 5132b01ca3ff05cb69861249a8609e0e893d0111
Author: Shalin Shekhar Mangar <sh...@apache.org>
AuthorDate: Wed Oct 25 20:26:33 2017 +0530
SOLR-11165: Ref guide documentation for Autoscaling APIs, triggers, listeners, actions for Solr 7.1
(cherry picked from commit 15e2729)
(cherry picked from commit 24e0f4e)
---
solr/CHANGES.txt | 9 ++--
.../src/solrcloud-autoscaling-actions.adoc | 28 ----------
.../solrcloud-autoscaling-auto-add-replicas.adoc | 59 +++++++++++++++++++++-
.../src/solrcloud-autoscaling-listeners.adoc | 55 +++++++++++++-------
.../src/solrcloud-autoscaling-overview.adoc | 16 ++++--
.../src/solrcloud-autoscaling-trigger-actions.adoc | 46 +++++++++++++++++
.../src/solrcloud-autoscaling-triggers.adoc | 32 +++++++-----
solr/solr-ref-guide/src/solrcloud-autoscaling.adoc | 6 ++-
8 files changed, 179 insertions(+), 72 deletions(-)
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index c7776d1..28c399e 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -70,7 +70,7 @@ Upgrade Notes
* SOLR-11254: the abstract DocTransformer class now has an abstract score-less transform method variant.
* SOLR-11283: all Stream Evaluators in solrj.io.eval have been refactored to have a simplier and more
- robust structure. This simplifies and condenses the code required to implement a new Evaluator and
+ robust structure. This simplifies and condenses the code required to implement a new Evaluator and
makes it much easier for evaluators to handle differing data types (primitives, objects, arrays,
lists, and so forth). (Dennis Gove)
@@ -298,10 +298,10 @@ Other Changes
* SOLR-11106: TestLBHttpSolrClient.testReliablity takes 30 seconds because of the wrong server name
(Kensho Hirasawa via Erick Erickson)
-
+
* SOLR-10338: Configure SecureRandom non blocking for tests. (Mihaly Toth, hossman, Ishan Chattopadhyaya, via Mark Miller)
-* SOLR-10916: Convert tests that extend LuceneTestCase and use MiniSolrCloudCluster
+* SOLR-10916: Convert tests that extend LuceneTestCase and use MiniSolrCloudCluster
to instead extend SolrCloudTestCase. (Steve Rowe)
* SOLR-11131: Document 'assert' as a command option in bin/solr, and bin/solr.cmd scripts.
@@ -360,6 +360,9 @@ Other Changes
* SOLR-10335: Upgrade to Tika 1.16. (Tim Allison, Tomás Fernández Löbbe via shalin)
+* SOLR-11165: Ref guide documentation for Autoscaling APIs, triggers, listeners, actions for Solr 7.1
+ (ab, Cassandra Targett, Noble Paul, shalin)
+
================== 7.0.1 ==================
Consult the LUCENE_CHANGES.txt file for additional, low level, changes in this release.
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling-actions.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling-actions.adoc
deleted file mode 100644
index 676527f..0000000
--- a/solr/solr-ref-guide/src/solrcloud-autoscaling-actions.adoc
+++ /dev/null
@@ -1,28 +0,0 @@
-= SolrCloud Autoscaling Actions
-:page-shortname: solrcloud-autoscaling-actions
-:page-permalink: solrcloud-autoscaling-actions.html
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-`TriggerAction` implementations process events generated by triggers in order to ensure the cluster's
-health and good use of resources.
-
-Currently two implementations are provided: `ComputePlanAction` and `ExecutePlanAction`.
-
-== Compute plan action
-
-== Execute plan action
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling-auto-add-replicas.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling-auto-add-replicas.adoc
index d2a027e..d5c9716 100644
--- a/solr/solr-ref-guide/src/solrcloud-autoscaling-auto-add-replicas.adoc
+++ b/solr/solr-ref-guide/src/solrcloud-autoscaling-auto-add-replicas.adoc
@@ -1,4 +1,4 @@
-= SolrCloud AutoScaling Automatically Adding Replicas
+= SolrCloud Autoscaling Automatically Adding Replicas
:page-shortname: solrcloud-autoscaling-auto-add-replicas
:page-permalink: solrcloud-autoscaling-auto-add-replicas.html
// Licensed to the Apache Software Foundation (ASF) under one
@@ -18,4 +18,59 @@
// specific language governing permissions and limitations
// under the License.
-TODO
\ No newline at end of file
+Solr provides a way to automatically add replicas for a collection when the number of active replicas drops below
+the replication factor specified at the time of the creation of the collection.
+
+== `autoAddReplicas` parameter
+
+The boolean `autoAddReplicas` parameter can be passed to the Create Collection API to enable this feature for a given collection.
+
+.Creating a collection with autoAddReplicas feature
+`http://localhost:8983/solr/admin/collections?action=CREATE&name=my_collection&numShards=1&replicationFactor=5&autoAddReplicas=true`
+
+The modify collection API can be used to enable/disable this feature for any collection.
+
+.Modifying collection to disable autoAddReplicas feature
+`http://localhost:8983/solr/admin/collections?action=MODIFYCOLLECTION&name=my_collection&autoAddReplicas=false`
+
+== Implementation using `.autoAddReplicas` trigger
+
+A Trigger named `.autoAddReplicas` is automatically created whenever any collection has the autoAddReplicas feature enabled.
+Only one trigger is sufficient to server all collections having this feature enabled. The `.autoAddReplicas` trigger watches
+for nodes that are lost from the cluster and uses the default TriggerActions to create new replicas to replace the ones
+which were hosted by the lost node. If the old node comes back online, it unloads the moved replicas and is free to host other
+replicas as and when required.
+
+Since the trigger provides the autoAddReplicas feature for all collections, the suspend-trigger and resume-trigger APIs
+can be used to disable and enable this feature for all collections in one API call.
+
+.Suspending autoAddReplicas for all collections
+[source,json]
+----
+{
+ "suspend-trigger": {
+ "name" : ".autoAddReplicas"
+ }
+}
+----
+
+.Resuming autoAddReplicas for all collections
+[source,json]
+----
+{
+ "resume-trigger": {
+ "name" : ".autoAddReplicas"
+ }
+}
+----
+
+== Using cluster property to enable/disable autoAddReplicas
+
+A cluster property, also named `autoAddReplicas`, can be set to `false` to disable this feature for all collections.
+If this cluster property is missing or set to `true` the autoAddReplicas is enabled for all collections.
+
+.Deprecation Warning
+[WARNING]
+====
+Using cluster property to enable/disable autoAddReplicas is deprecated and only supported for back compatibility. Please use the suspend-trigger and resume-trigger APIs instead.
+====
\ No newline at end of file
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling-listeners.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling-listeners.adoc
index e4a3f55..282839a 100644
--- a/solr/solr-ref-guide/src/solrcloud-autoscaling-listeners.adoc
+++ b/solr/solr-ref-guide/src/solrcloud-autoscaling-listeners.adoc
@@ -1,4 +1,4 @@
-= SolrCloud AutoScaling Listeners
+= SolrCloud Autoscaling Listeners
:page-shortname: solrcloud-autoscaling-listeners
:page-permalink: solrcloud-autoscaling-listeners.html
// Licensed to the Apache Software Foundation (ASF) under one
@@ -67,11 +67,11 @@ For example:
[source,json]
----
{
- 'set-listener': {
- 'name': 'foo',
- 'trigger': 'node_lost_trigger',
- 'stage': ['STARTED', 'ABORTED', 'SUCCEEDED', 'FAILED'],
- 'class': 'solr.SystemLogListener'
+ "set-listener": {
+ "name": "foo",
+ "trigger": "node_lost_trigger",
+ "stage": ["STARTED", "ABORTED", "SUCCEEDED", "FAILED"],
+ "class": "solr.SystemLogListener"
}
}
----
@@ -79,8 +79,8 @@ For example:
[source,json]
----
{
- 'remove-listener': {
- 'name': 'foo'
+ "remove-listener": {
+ "name": "foo"
}
}
----
@@ -95,7 +95,12 @@ below, together with their configuration parameters.
This trigger listener sends trigger events and processing context as documents for indexing in
SolrCloud `.system` collection.
+When a trigger configuration is first created a corresponding trigger listener configuration that
+uses `SystemLogListener` is also automatically created, to make sure that all events and
+actions related to the autoscaling framework are logged to the `.system` collection.
+
Supported configuration properties:
+
* `collection` - (string, optional) specifies the target collection where documents are sent.
Default value is `.system`
* `enabled` - (boolean, optional) enables the listener when true. Default value is true.
@@ -128,6 +133,18 @@ processed)
`_s` or `_ss` suffix depending on whether the property value is a collection (values inside collection are treated as
strings, there's no recursive flattening)
+The following configuration is used for the automatically created listener (in this case for a
+trigger named `foo`):
+
+[source,json]
+----
+{
+ "name" : "foo.system",
+ "trigger" : "solr.SystemLogListener",
+ "stage" : ["WAITING", "STARTED", "ABORTED", "SUCCEEDED", "FAILED", "BEFORE_ACTION", "AFTER_ACTION"]
+}
+----
+
=== `HttpTriggerListener`
This listener uses HTTP POST to send a representation of event and context to a specified URL.
URL, payload and headers may contain property substitution patterns, which are then replaced with values takes from the
@@ -160,17 +177,17 @@ Example configuration:
[source,json]
----
{
- 'name': 'foo',
- 'trigger': 'node_added_trigger',
- 'class': 'solr.HttpTriggerListener',
- 'url': 'http://foo.com/${config.name:invalidName}/${config.properties.xyz:invalidXyz}/${event.eventType}',
- 'xyz': 'foobar',
- 'header.X-Trigger': '${config.trigger}',
- 'payload': 'actionName=${actionName}, source=${event.source}, type=${event.eventType}',
- 'contentType': 'text/plain',
- 'stage': ['STARTED', 'ABORTED', SUCCEEDED', 'FAILED'],
- 'beforeAction': ['compute_plan', 'execute_plan'],
- 'afterAction': ['compute_plan', 'execute_plan']
+ "name": "foo",
+ "trigger": "node_added_trigger",
+ "class": "solr.HttpTriggerListener",
+ "url": "http://foo.com/${config.name:invalidName}/${config.properties.xyz:invalidXyz}/${event.eventType}",
+ "xyz": "foobar",
+ "header.X-Trigger": "${config.trigger}",
+ "payload": "actionName=${actionName}, source=${event.source}, type=${event.eventType}",
+ "contentType": "text/plain",
+ "stage": ["STARTED", "ABORTED", SUCCEEDED", "FAILED"],
+ "beforeAction": ["compute_plan", "execute_plan"],
+ "afterAction": ["compute_plan", "execute_plan"]
}
----
This configuration specifies that each time one of the listed stages is reached, or before and after each of the listed
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc
index 4ea7612..b3f2763 100644
--- a/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc
+++ b/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc
@@ -44,7 +44,7 @@ We have an easy way to enable this behavior without needing to understand the au
A collection created with `autoAddReplicas=true` will be monitored by Solr such that if a node containing a replica of this collection goes down, Solr will add new replicas on other nodes after waiting for up to thirty seconds for the node to come back.
-You can see the section __TODO_FIX_ME__ to learn more about how to enable or disable this feature as well as other details.
+You can see the section <<solrcloud-autoscaling-auto-add-replicas.adoc#solrcloud-autoscaling-auto-add-replicas,SolrCloud AutoScaling Automatically Adding Replicas>> to learn more about how to enable or disable this feature as well as other details.
The selection of the node that will host the new replica is made according to the default cluster preferences that we will learn more about in the next sections.
@@ -83,15 +83,21 @@ You can learn more about collection-specific policies in the section <<solrclou
Now that we have an idea about how cluster management operations use policy and preferences help Solr keep the cluster balanced and stable, we can talk about when to invoke such operations. Triggers are used to watch for events such as a node joining or leaving the cluster. When the event happens, the trigger executes a set of `actions` that compute and execute a *plan* i.e. a set of operations to change the cluster so that the policy and preferences are respected.
-The `autoAddReplicas` parameter passed to the create collection API in the quickstart section automatically creates a trigger that watches for a node going away. When the trigger fires, it computes and executes a plan to move all replicas hosted by the lost node to new nodes in the cluster. The target nodes are chosen based on the policy and preferences.
+The `autoAddReplicas` parameter passed to the create collection API in the quickstart section automatically creates a trigger that watches for a node going away. When the trigger fires, it executes a set of actions that compute and execute a plan to move all replicas hosted by the lost node to new nodes in the cluster. The target nodes are chosen based on the policy and preferences.
-You can learn more about Triggers in the __TODO__ section.
+You can learn more about Triggers in the section <<solrcloud-autoscaling-triggers.adoc#solrcloud-autoscaling-triggers,Autoscaling Triggers>>.
+
+== Trigger Actions
+
+A trigger executes *actions* that tell Solr what to do in response to the trigger. Solr ships with two actions that are added to every trigger by default. The first is called the *ComputePlanAction* and the other is *ExecutePlanAction*. The former computes the cluster management operations necessary to stabilize the cluster and the latter executes them on the cluster.
+
+You can learn more about Trigger Actions in the section <<solrcloud-autoscaling-trigger-actions.adoc#solrcloud-autoscaling-trigger-actions,Autoscaling Trigger Actions>>.
== Listeners
-An AutoScaling *Listener* is attached to a trigger. Solr calls the listener each time the trigger fires as well as before and after the actions performed by the trigger. Listeners are useful as a call back mechanism to perform tasks such as logging or informing external systems about events. For example, a listener is automatically added by Solr to each Trigger to log details of the trigger fire and actions to the `.system` collection.
+An AutoScaling *Listener* can be attached to a trigger. Solr calls the listener each time the trigger fires as well as before and after the actions performed by the trigger. Listeners are useful as a call back mechanism to perform tasks such as logging or informing external systems about events. For example, a listener is automatically added by Solr to each Trigger to log details of the trigger fire and actions to the `.system` collection.
-You can learn more about Listeners in the __TODO__ section.
+You can learn more about Listeners in the section <<solrcloud-autoscaling-policy-preferences.adoc#solrcloud-autoscaling-policy-preferences,AutoScaling Listeners>>.
== Autoscaling APIs
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling-trigger-actions.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling-trigger-actions.adoc
new file mode 100644
index 0000000..89dfd17
--- /dev/null
+++ b/solr/solr-ref-guide/src/solrcloud-autoscaling-trigger-actions.adoc
@@ -0,0 +1,46 @@
+= SolrCloud Autoscaling Trigger Actions
+:page-shortname: solrcloud-autoscaling-trigger-actions
+:page-permalink: solrcloud-autoscaling-trigger-actions.html
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+`TriggerAction` implementations process events generated by triggers in order to ensure the cluster's
+health and good use of resources.
+
+Currently two implementations are provided: `ComputePlanAction` and `ExecutePlanAction`.
+
+== Compute plan action
+
+The `ComputePlanAction` uses the policy and preferences to calculate the optimal set of Collection API
+commands which can re-balance the cluster in response to trigger events.
+
+Currently, it has no configurable parameters.
+
+== Execute plan action
+
+The `ExecutePlanAction` executes the Collection API commands emitted by the `ComputePlanAction` against
+the cluster using SolrJ. It executes the commands serially, waiting for each of them to succeed before
+continuing with the next one.
+
+Currently, it has no configurable parameters.
+
+If any one of the command fails, then the complete chain of actions are
+executed again at the next run of the trigger. If the Overseer node fails while ExecutePlanAction is running
+then the new overseer node will run the chain of actions for the same event again after waiting for any
+running Collection API operations belonging to the event to complete.
+
+Please see <<solrcloud-autoscaling-fault-tolerance.adoc#solrcloud-autoscaling-fault-tolerance,SolrCloud AutoScaling Fault Tolerance>> for more details on fault tolerance within the Autoscaling framework.
\ No newline at end of file
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling-triggers.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling-triggers.adoc
index 38945f1..0893166 100644
--- a/solr/solr-ref-guide/src/solrcloud-autoscaling-triggers.adoc
+++ b/solr/solr-ref-guide/src/solrcloud-autoscaling-triggers.adoc
@@ -1,4 +1,4 @@
-= SolrCloud AutoScaling Triggers
+= SolrCloud Autoscaling Triggers
:page-shortname: solrcloud-autoscaling-triggers
:page-permalink: solrcloud-autoscaling-triggers.html
// Licensed to the Apache Software Foundation (ASF) under one
@@ -17,6 +17,7 @@
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
+
Triggers are used by autoscaling API to watch for cluster events such as node joining or leaving,
and in the future also for other cluster, node and replica events that are important from the
point of view of cluster performance.
@@ -55,6 +56,8 @@ is automatically created to watch for nodes going away. This trigger produces `n
which are then processed by configured actions (usually resulting in computing and executing a plan
to add replicas on the live nodes to maintain the expected replication factor).
+You can see the section <<solrcloud-autoscaling-auto-add-replicas.adoc#solrcloud-autoscaling-auto-add-replicas,SolrCloud AutoScaling Automatically Adding Replicas>> to learn more about how `.autoAddReplicas` work.
+
== Trigger configuration
Trigger configurations are managed using autoscaling Write API with commands `set-trigger`, `remove-trigger`,
`suspend-trigger`, `resume-trigger`.
@@ -75,29 +78,32 @@ Action configuration consists of the following properties:
* `class` - (string, required) action implementation class
* additional implementation-specific properties may be provided
+If the Action configuration is omitted, then by default, the ComputePlanAction and the ExecutePlanAction are automatically
+added to the trigger configuration.
+
Example: adding / updating a trigger for `nodeAdded` events. This trigger configuration will
compute and execute a plan to allocate the resources available on the new node. A custom action
is also used to possibly modify the plan.
[source,json]
----
{
- 'set-trigger': {
- 'name' : 'node_lost_trigger',
- 'event' : 'nodeLost',
- 'waitFor' : '1s',
- 'enabled' : true,
- 'actions' : [
+ "set-trigger": {
+ "name" : "node_added_trigger",
+ "event" : "nodeAdded",
+ "waitFor" : "1s",
+ "enabled" : true,
+ "actions" : [
{
- 'name' : 'compute_plan',
- 'class': 'solr.ComputePlanAction'
+ "name" : "compute_plan",
+ "class": "solr.ComputePlanAction"
},
{
- 'name' : 'custom_action',
- 'class': 'com.example.CustomAction'
+ "name" : "custom_action",
+ "class": "com.example.CustomAction"
},
{
- 'name' : 'execute_plan',
- 'class': 'solr.ExecutePlanAction'
+ "name" : "execute_plan",
+ "class": "solr.ExecutePlanAction"
}
]
}
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling.adoc
index 3c0d2c5..5d53b6e 100644
--- a/solr/solr-ref-guide/src/solrcloud-autoscaling.adoc
+++ b/solr/solr-ref-guide/src/solrcloud-autoscaling.adoc
@@ -1,7 +1,7 @@
= SolrCloud AutoScaling
:page-shortname: solrcloud-autoscaling
:page-permalink: solrcloud-autoscaling.html
-:page-children: solrcloud-autoscaling-overview, solrcloud-autoscaling-api, solrcloud-autoscaling-policy-preferences, solrcloud-autoscaling-triggers, solrcloud-autoscaling-listeners, solrcloud-autoscaling-auto-add-replicas
+:page-children: solrcloud-autoscaling-overview, solrcloud-autoscaling-api, solrcloud-autoscaling-policy-preferences, solrcloud-autoscaling-triggers, solrcloud-autoscaling-trigger-actions, solrcloud-autoscaling-listeners, solrcloud-autoscaling-auto-add-replicas, solrcloud-autoscaling-fault-tolerance
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
@@ -22,7 +22,7 @@
[.lead]
The goal of autoscaling is to make SolrCloud cluster management easier by providing a way for changes to the cluster to be more automatic and more intelligent.
-Autoscaling includes an API to manage cluster-wide and collection-specific policies and preferences and a rules syntax to define the guidelines for your cluster. Future Solr releases will include features to utilize the policies and preferences so they perform actions automatically when the rules are violated.
+Autoscaling includes an API to manage cluster-wide and collection-specific policies and preferences and a rules syntax to define the guidelines for your cluster. Also included are features to utilize the policies and preferences so they perform actions automatically when certain conditions are met.
The following sections describe the autoscaling features of SolrCloud:
@@ -30,5 +30,7 @@ The following sections describe the autoscaling features of SolrCloud:
* <<solrcloud-autoscaling-api.adoc#solrcloud-autoscaling-api,SolrCloud Autoscaling API>>
* <<solrcloud-autoscaling-policy-preferences.adoc#solrcloud-autoscaling-policy-preferences,SolrCloud Autoscaling Policy and Preferences>>
* <<solrcloud-autoscaling-triggers.adoc#solrcloud-autoscaling-triggers,SolrCloud AutoScaling Triggers>>
+* <<solrcloud-autoscaling-trigger-actions.adoc#solrcloud-autoscaling-trigger-actions,SolrCloud AutoScaling Trigger Actions>>
* <<solrcloud-autoscaling-listeners.adoc#solrcloud-autoscaling-listeners,SolrCloud AutoScaling Listeners>>
* <<solrcloud-autoscaling-auto-add-replicas.adoc#solrcloud-autoscaling-auto-add-replicas,SolrCloud AutoScaling - Automatically Adding Replicas>>
+* <<solrcloud-autoscaling-fault-tolerance.adoc#solrcloud-autoscaling-fault-tolerance,SolrCloud AutoScaling Fault Tolerance>>