You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2021/01/18 14:41:57 UTC
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #2215: SOLR-14067: v3 Create /contrib/scripting module with ScriptingUpdateProcessor
dsmiley commented on a change in pull request #2215:
URL: https://github.com/apache/lucene-solr/pull/2215#discussion_r559588597
##########
File path: solr/contrib/scripting/README.md
##########
@@ -0,0 +1,14 @@
+Welcome to Apache Solr Scripting!
+===============================
+
+# Introduction
+
+The Scripting contrib module pulls together various scripting related functions.
+
+Today, the ScriptUpdateProcessorFactory allows Java scripting engines to be used during the Solr document update processing, allowing dramatic flexibility in expressing custom document processing before being indexed. It also allows hooks to commit, delete, etc, but add is the most common usage. It is implemented as an UpdateProcessor to be placed in an UpdateChain.
Review comment:
Lets list a few popular options here -- I'm thinking JavaScript, Ruby, Python, Groovy
##########
File path: solr/contrib/scripting/src/java/org/apache/solr/scripting/update/ScriptUpdateProcessorFactory.java
##########
@@ -58,34 +60,34 @@
/**
* <p>
- * An update request processor factory that enables the use of update
- * processors implemented as scripts which can be loaded by the
- * {@link SolrResourceLoader} (usually via the <code>conf</code> dir for
- * the SolrCore).
+ * An update request processor factory that enables the use of update
+ * processors implemented as scripts which can be loaded by the
+ * {@link SolrResourceLoader} (usually via the <code>conf</code> dir for
+ * the SolrCore). Previously known as the StatelessScriptUpdateProcessor.
Review comment:
```suggestion
* processors implemented as scripts which can be loaded from the
* configSet. Previously known as the StatelessScriptUpdateProcessor.
```
##########
File path: solr/solr-ref-guide/src/scripting-update-processor.adoc
##########
@@ -0,0 +1,295 @@
+= Scripting Update Processor
Review comment:
```suggestion
= Script Update Processor
```
And Can we rename this file to remove the "ing"?
The PR shows this file as new; did you just write all this?
##########
File path: solr/solr-ref-guide/src/scripting-update-processor.adoc
##########
@@ -0,0 +1,295 @@
+= Scripting Update Processor
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The {solr-javadocs}/contrib/scripting/org/apache/solr/scripting/update/ScriptUpdateProcessorFactory.html[ScriptUpdateProcessor] allows Java scripting engines to be used
+during Solr document update processing, allowing dramatic flexibility in
+expressing custom document processing logic before being indexed. It has hooks to the
+commit, delete, rollback, etc indexing actions, however add is the most common usage.
+It is implemented as an UpdateProcessor to be placed in an UpdateChain.
+
+TIP: This used to be known as the _StatelessScriptingUpdateProcessor_ and was renamed to clarify the key aspect of this update processor is it enables scripting.
+
+The script can be written in any scripting language supported by your JVM (such
+as JavaScript), and executed dynamically so no pre-compilation is necessary.
+
+WARNING: Being able to run a script of your choice as part of the indexing pipeline is a really powerful tool, that I sometimes call the
+_Get out of jail free_ card because you can solve some problems this way that you can't in any other way. However, you are introducing some
+potential security vulnerabilities.
+
+== Installing the ScriptingUpdateProcessor and Scripting Engines
+
+The scripting update processor lives in the contrib module `/contrib/scripting`, and you need to explicitly add it to your Solr setup.
+
+Java 11 and previous versions come with a JavaScript engine called Nashorn, but Java 12 will require you to add your own JavaScript engine. Other supported scripting engines like
+JRuby, Jython, Groovy, all require you to add JAR files.
+
+
+You can either add the `dist/solr-scripting-*.jar` file into Solr’s resource loader in a core `lib/` directory, or via `<lib>` directives in `solrconfig.xml`:
+
+[source,xml]
+----
+<lib dir="${solr.install.dir:../../..}/dist/" regex="solr-scripting-\d.*\.jar" />
+----
+
+Likewise you will need to add some JAR files depending on which scripting engines you choose.
+
+
+== Configuration
+
+[source,xml]
+----
+<updateRequestProcessorChain name="script">
+ <processor class="org.apache.solr.scripting.update.ScriptUpdateProcessorFactory">
+ <str name="script">update-script.js</str>
+ </processor>
+ <!-- optional parameters passed to script
+ <lst name="params">
+ <str name="config_param">example config parameter</str>
+ </lst>
+ -->
+ <processor class="solr.LogUpdateProcessorFactory" />
+ <processor class="solr.RunUpdateProcessorFactory" />
+ </updateRequestProcessorChain>
+----
+
+NOTE: The processor supports the defaults/appends/invariants concept for its config.
+However, it is also possible to skip this level and configure the parameters directly underneath the `<processor>` tag.
+
+Below follows a list of each configuration parameters and their meaning:
+
+`script`::
+The script file name. The script file must be placed in the `conf/ directory.
+There can be one or more "script" parameters specified; multiple scripts are executed in the order specified.
+
+`engine`::
+Optionally specifies the scripting engine to use. This is only needed if the extension
+of the script file is not a standard mapping to the scripting engine. For example, if your
+script file was coded in JavaScript but the file name was called `update-script.foo`,
+use "javascript" as the engine name.
+
+`params`::
+Optional parameters that are passed into the script execution context. This is
+specified as a named list (`<lst>`) structure with nested typed parameters. If
+specified, the script context will get a "params" object, otherwise there will be no "params" object available.
+
+
+== Script execution context
+
+Every script has some variables provided to it.
+
+`logger`::
+Logger (org.slf4j.Logger) instance. This is useful for logging information from the script.
+
+`req`::
+{solr-javadocs}/core/org/apache/solr/response/SolrQueryResponse.html[SolrQueryRequest] instance.
+
+`rsp`::
+{solr-javadocs}/core/org/apache/solr/response/SolrQueryResponse.html[SolrQueryResponse] instance.
+
+`params`::
+The "params" object, if any specified, from the configuration.
+
+== Examples
+
+The `processAdd()` and the other script methods can return false to skip further
+processing of the document. All methods must be defined, though generally the
+`processAdd()` method is where the action is.
+
+Here's a URL that works with the techproducts example setup demonstrating specifying
+the "script" update chain: `http://localhost:8983/solr/techproducts/update?commit=true&stream.contentType=text/csv&fieldnames=id,description&stream.body=1,foo&update.chain=script`
+which logs the following:
+
+[source,text]
+----
+INFO: update-script#processAdd: id=1
+----
+
+You can see the message recorded in the Solr logging UI.
+
+=== Javascript
+
+Note: There is a JavaScript example `update-script.js` as part of the `techproducts` configset.
+Check `solrconfig.xml` and uncomment the update request processor definition to enable this feature.
+
+[source,javascript]
+----
+function processAdd(cmd) {
+
+ doc = cmd.solrDoc; // org.apache.solr.common.SolrInputDocument
+ id = doc.getFieldValue("id");
+ logger.info("update-script#processAdd: id=" + id);
+
+// Set a field value:
+// doc.setField("foo_s", "whatever");
+
+// Get a configuration parameter:
+// config_param = params.get('config_param'); // "params" only exists if processor configured with <lst name="params">
+
+// Get a request parameter:
+// some_param = req.getParams().get("some_param")
+
+// Add a field of field names that match a pattern:
+// - Potentially useful to determine the fields/attributes represented in a result set, via faceting on field_name_ss
+// field_names = doc.getFieldNames().toArray();
+// for(i=0; i < field_names.length; i++) {
+// field_name = field_names[i];
+// if (/attr_.*/.test(field_name)) { doc.addField("attribute_ss", field_names[i]); }
+// }
+
+}
+
+function processDelete(cmd) {
+ // no-op
+}
+
+function processMergeIndexes(cmd) {
+ // no-op
+}
+
+function processCommit(cmd) {
+ // no-op
+}
+
+function processRollback(cmd) {
+ // no-op
+}
+
+function finish() {
+ // no-op
+}
+----
+
+=== JRuby
+
+To use JRuby as the scripting engine, add `jruby.jar` to Solr's resource loader.
Review comment:
I think the term "resource loader" ought to be internal, and we should avoid using in documentation when we can be clearer. We can just say add to a lib directory? We could also link to the existing ref guide page on installing libs.
##########
File path: solr/contrib/scripting/src/java/org/apache/solr/scripting/update/package-info.java
##########
@@ -0,0 +1,21 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Support for scripting during document updates.
Review comment:
```suggestion
* Support for scripting during document updates.
```
##########
File path: solr/CHANGES.txt
##########
@@ -186,6 +186,9 @@ Other Changes
* SOLR-14034: Remove deprecated min_rf references (Tim Dillon)
+* SOLR-14067: StatelessScriptUpdateProcessor moved to it's own /contrib/scripting/ package instead
+ of shipping as part of Solr due to security concerns. Renamed to ScriptingUpdateProcessor. (Eric Pugh)
Review comment:
```suggestion
of shipping as part of Solr due to security concerns. Renamed to ScriptUpdateProcessorFactory. (Eric Pugh)
```
##########
File path: solr/server/solr/configsets/sample_techproducts_configs/conf/solrconfig.xml
##########
@@ -674,12 +679,12 @@
*** WARNING ***
Before enabling remote streaming, you should make sure your
system has authentication enabled.
-
- <requestParsers enableRemoteStreaming="false"
+ -->
+ <requestParsers enableRemoteStreaming="true"
Review comment:
Why?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org