You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ja...@apache.org on 2017/07/06 14:34:38 UTC
[1/2] lucene-solr:branch_7_0: SOLR-10574,
SOLR-10272: Refguide documentation for _default configset
Repository: lucene-solr
Updated Branches:
refs/heads/branch_7_0 7dfc0d4ef -> 257883d65
SOLR-10574, SOLR-10272: Refguide documentation for _default configset
(cherry picked from commit 112bdda)
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/04054fc5
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/04054fc5
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/04054fc5
Branch: refs/heads/branch_7_0
Commit: 04054fc524432388c7cea722b766815a950ca736
Parents: 7dfc0d4
Author: Ishan Chattopadhyaya <is...@apache.org>
Authored: Thu Jul 6 14:24:02 2017 +0200
Committer: Jan Høydahl <ja...@apache.org>
Committed: Thu Jul 6 16:23:45 2017 +0200
----------------------------------------------------------------------
.../src/command-line-utilities.adoc | 2 +-
solr/solr-ref-guide/src/enabling-ssl.adoc | 2 +-
.../src/getting-started-with-solrcloud.adoc | 4 +++-
.../src/initparams-in-solrconfig.adoc | 2 +-
solr/solr-ref-guide/src/schemaless-mode.adoc | 11 +++--------
.../src/solr-control-script-reference.adoc | 18 ++++++++++--------
...ing-data-with-solr-cell-using-apache-tika.adoc | 2 +-
...g-zookeeper-to-manage-configuration-files.adoc | 6 +++---
.../src/velocity-response-writer.adoc | 2 +-
9 files changed, 24 insertions(+), 25 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/04054fc5/solr/solr-ref-guide/src/command-line-utilities.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/command-line-utilities.adoc b/solr/solr-ref-guide/src/command-line-utilities.adoc
index 68d1385..e927f02 100644
--- a/solr/solr-ref-guide/src/command-line-utilities.adoc
+++ b/solr/solr-ref-guide/src/command-line-utilities.adoc
@@ -103,7 +103,7 @@ If you are on Windows machine, simply replace `zkcli.sh` with `zkcli.bat` in the
[source,bash]
----
-./server/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:9983 -cmd upconfig -confname my_new_config -confdir server/solr/configsets/basic_configs/conf
+./server/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:9983 -cmd upconfig -confname my_new_config -confdir server/solr/configsets/_default/conf
----
[[CommandLineUtilities-BootstrapZooKeeperfromexistingSOLR_HOME]]
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/04054fc5/solr/solr-ref-guide/src/enabling-ssl.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/enabling-ssl.adoc b/solr/solr-ref-guide/src/enabling-ssl.adoc
index 32b3cc0..be2025e 100644
--- a/solr/solr-ref-guide/src/enabling-ssl.adoc
+++ b/solr/solr-ref-guide/src/enabling-ssl.adoc
@@ -255,7 +255,7 @@ NOTE: If your operating system does not include cURL, you can download binaries
=== Create a SolrCloud Collection using bin/solr
-Create a 2-shard, replicationFactor=1 collection named mycollection using the default configset (data_driven_schema_configs):
+Create a 2-shard, replicationFactor=1 collection named mycollection using the default configset (_default):
.*nix command
[source,bash]
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/04054fc5/solr/solr-ref-guide/src/getting-started-with-solrcloud.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/getting-started-with-solrcloud.adoc b/solr/solr-ref-guide/src/getting-started-with-solrcloud.adoc
index b07901d..d512660 100644
--- a/solr/solr-ref-guide/src/getting-started-with-solrcloud.adoc
+++ b/solr/solr-ref-guide/src/getting-started-with-solrcloud.adoc
@@ -89,7 +89,9 @@ Next, the script prompts you for the number of shards to distribute the collecti
Next, the script will prompt you for the number of replicas to create for each shard. <<shards-and-indexing-data-in-solrcloud.adoc#shards-and-indexing-data-in-solrcloud,Replication>> is covered in more detail later in the guide, so if you're unsure, then use the default of 2 so that you can see how replication is handled in SolrCloud.
-Lastly, the script will prompt you for the name of a configuration directory for your collection. You can choose *basic_configs*, *data_driven_schema_configs*, or *sample_techproducts_configs*. The configuration directories are pulled from `server/solr/configsets/` so you can review them beforehand if you wish. The *data_driven_schema_configs* configuration (the default) is useful when you're still designing a schema for your documents and need some flexiblity as you experiment with Solr.
+Lastly, the script will prompt you for the name of a configuration directory for your collection. You can choose *_default*, or *sample_techproducts_configs*. The configuration directories are pulled from `server/solr/configsets/` so you can review them beforehand if you wish. The *_default* configuration is useful when you're still designing a schema for your documents and need some flexiblity as you experiment with Solr, since it has schemaless functionality. However, after creating your collection, the schemaless functionality can be disabled in order to lock down the schema (so that documents indexed after doing so will not alter the schema) or to configure the schema by yourself. This can be done as follows (assuming your collection name is `mycollection`):
+
+`curl http://host:8983/solr/mycollection/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'`
At this point, you should have a new collection created in your local SolrCloud cluster. To verify this, you can run the status command:
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/04054fc5/solr/solr-ref-guide/src/initparams-in-solrconfig.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/initparams-in-solrconfig.adoc b/solr/solr-ref-guide/src/initparams-in-solrconfig.adoc
index ac409ff..1120e43 100644
--- a/solr/solr-ref-guide/src/initparams-in-solrconfig.adoc
+++ b/solr/solr-ref-guide/src/initparams-in-solrconfig.adoc
@@ -29,7 +29,7 @@ For example, if you want several of your search handlers to return the same list
The properties and configuration of an `<initParams>` section mirror the properties and configuration of a request handler. It can include sections for defaults, appends, and invariants, the same as any request handler.
-For example, here is one of the `<initParams>` sections defined by default in the `data_driven_config` example:
+For example, here is one of the `<initParams>` sections defined by default in the `_default` example:
[source,xml]
----
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/04054fc5/solr/solr-ref-guide/src/schemaless-mode.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/schemaless-mode.adoc b/solr/solr-ref-guide/src/schemaless-mode.adoc
index 45ae8dd..bdc1888 100644
--- a/solr/solr-ref-guide/src/schemaless-mode.adoc
+++ b/solr/solr-ref-guide/src/schemaless-mode.adoc
@@ -29,7 +29,7 @@ These Solr features, all controlled via `solrconfig.xml`, are:
[[SchemalessMode-UsingtheSchemalessExample]]
== Using the Schemaless Example
-The three features of schemaless mode are pre-configured in the `data_driven_schema_configs` <<config-sets.adoc#config-sets,config set>> in the Solr distribution. To start an example instance of Solr using these configs, run the following command:
+The three features of schemaless mode are pre-configured in the `_default` <<config-sets.adoc#config-sets,config set>> in the Solr distribution. To start an example instance of Solr using these configs, run the following command:
[source,bash]
----
@@ -67,15 +67,10 @@ You can use the `/schema/fields` <<schema-api.adoc#schema-api,Schema API>> to co
"uniqueKey":true}]}
----
-[TIP]
-====
-The `data_driven_schema_configs` configset includes a `copyField` directive that causes all content to be indexed in a predefined "catch-all" `\_text_` field, which is used to enable single-field search that includes all fields' content. This will cause the index to be larger than it would be without this "catch-all" `copyField`. When you nail down your schema, consider removing the `\_text_` field and the corresponding `copyField` directive if you don't need it.
-====
-
[[SchemalessMode-ConfiguringSchemalessMode]]
== Configuring Schemaless Mode
-As described above, there are three configuration elements that need to be in place to use Solr in schemaless mode. In the `data_driven_schema_configs` config set included with Solr these are already configured. If, however, you would like to implement schemaless on your own, you should make the following changes.
+As described above, there are three configuration elements that need to be in place to use Solr in schemaless mode. In the default (`_default`) config set included with Solr these are already configured. If, however, you would like to implement schemaless on your own, you should make the following changes.
[[SchemalessMode-EnableManagedSchema]]
=== Enable Managed Schema
@@ -190,7 +185,7 @@ After each of these changes have been made, Solr should be restarted (or, you ca
[[SchemalessMode-ExamplesofIndexedDocuments]]
== Examples of Indexed Documents
-Once the schemaless mode has been enabled (whether you configured it manually or are using `data_driven_schema_configs` ), documents that include fields that are not defined in your schema should be added to the index, and the new fields added to the schema.
+Once the schemaless mode has been enabled (whether you configured it manually or are using `_default` ), documents that include fields that are not defined in your schema should be added to the index, and the new fields added to the schema.
For example, adding a CSV document will cause its fields that are not in the schema to be added, with fieldTypes based on values:
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/04054fc5/solr/solr-ref-guide/src/solr-control-script-reference.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/solr-control-script-reference.adoc b/solr/solr-ref-guide/src/solr-control-script-reference.adoc
index c933114..45a9e80 100644
--- a/solr/solr-ref-guide/src/solr-control-script-reference.adoc
+++ b/solr/solr-ref-guide/src/solr-control-script-reference.adoc
@@ -213,7 +213,7 @@ The configset used is customized for DIH, and is found in `$SOLR_HOME/example/ex
For more information about DIH, see the section <<uploading-structured-data-store-data-with-the-data-import-handler.adoc#uploading-structured-data-store-data-with-the-data-import-handler,Uploading Structured Data Store Data with the Data Import Handler>>.
* *schemaless*: This example starts Solr in standalone mode using a managed schema, as described in the section <<schema-factory-definition-in-solrconfig.adoc#schema-factory-definition-in-solrconfig,Schema Factory Definition in SolrConfig>>, and provides a very minimal pre-defined schema. Solr will run in <<schemaless-mode.adoc#schemaless-mode,Schemaless Mode>> with this configuration, where Solr will create fields in the schema on the fly and will guess field types used in incoming documents.
+
-The configset used can be found in `$SOLR_HOME/server/solr/configsets/data_driven_schema_configs`.
+The configset used can be found in `$SOLR_HOME/server/solr/configsets/_default`.
[IMPORTANT]
====
@@ -392,11 +392,11 @@ Name of the core or collection to create (required).
*Example*: `bin/solr create -c mycollection`
`-d <confdir>`::
-The configuration directory. This defaults to `data_driven_schema_configs`.
+The configuration directory. This defaults to `_default`.
+
See the section <<Configuration Directories and SolrCloud>> below for more details about this option when running in SolrCloud mode.
+
-*Example*: `bin/solr create -d basic_configs`
+*Example*: `bin/solr create -d _default`
`-n <configName>`::
The configuration name. This defaults to the same name as the core or collection.
@@ -431,15 +431,15 @@ Before creating a collection in SolrCloud, the configuration directory used by t
Let's work through a few examples to illustrate how configuration directories work in SolrCloud.
-First, if you don't provide the `-d` or `-n` options, then the default configuration (`$SOLR_HOME/server/solr/configsets/data_driven_schema_configs/conf`) is uploaded to ZooKeeper using the same name as the collection.
+First, if you don't provide the `-d` or `-n` options, then the default configuration (`$SOLR_HOME/server/solr/configsets/_default/conf`) is uploaded to ZooKeeper using the same name as the collection.
-For example, the following command will result in the `data_driven_schema_configs` configuration being uploaded to `/configs/contacts` in ZooKeeper: `bin/solr create -c contacts`.
+For example, the following command will result in the `_default` configuration being uploaded to `/configs/contacts` in ZooKeeper: `bin/solr create -c contacts`.
-If you create another collection with `bin/solr create -c contacts2`, then another copy of the `data_driven_schema_configs` directory will be uploaded to ZooKeeper under `/configs/contacts2`.
+If you create another collection with `bin/solr create -c contacts2`, then another copy of the `_default` directory will be uploaded to ZooKeeper under `/configs/contacts2`.
Any changes you make to the configuration for the contacts collection will not affect the `contacts2` collection. Put simply, the default behavior creates a unique copy of the configuration directory for each collection you create.
-You can override the name given to the configuration directory in ZooKeeper by using the `-n` option. For instance, the command `bin/solr create -c logs -d basic_configs -n basic` will upload the `server/solr/configsets/basic_configs/conf` directory to ZooKeeper as `/configs/basic`.
+You can override the name given to the configuration directory in ZooKeeper by using the `-n` option. For instance, the command `bin/solr create -c logs -d _default -n basic` will upload the `server/solr/configsets/_default/conf` directory to ZooKeeper as `/configs/basic`.
Notice that we used the `-d` option to specify a different configuration than the default. Solr provides several built-in configurations under `server/solr/configsets`. However you can also provide the path to your own configuration directory using the `-d` option. For instance, the command `bin/solr create -c mycoll -d /tmp/myconfigs`, will upload `/tmp/myconfigs` into ZooKeeper under `/configs/mycoll` .
@@ -449,7 +449,9 @@ Other collections can share the same configuration by specifying the name of the
==== Data-driven Schema and Shared Configurations
-The `data_driven_schema_configs` schema can mutate as data is indexed. Consequently, we recommend that you do not share data-driven configurations between collections unless you are certain that all collections should inherit the changes made when indexing data into one of the collections.
+The `_default` schema can mutate as data is indexed, since it has schemaless functionality (i.e. data-driven changes to the schema). Consequently, we recommend that you do not share data-driven configurations between collections unless you are certain that all collections should inherit the changes made when indexing data into one of the collections. You can turn off schemaless functionality (i.e. data-driven changes to the schema) for a collection by the following (assuming the collection name is `mycollection`):
+
+`curl http://host:8983/solr/mycollection/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'`
=== Delete Core or Collection
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/04054fc5/solr/solr-ref-guide/src/uploading-data-with-solr-cell-using-apache-tika.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/uploading-data-with-solr-cell-using-apache-tika.adoc b/solr/solr-ref-guide/src/uploading-data-with-solr-cell-using-apache-tika.adoc
index 894e949..cdd9539 100644
--- a/solr/solr-ref-guide/src/uploading-data-with-solr-cell-using-apache-tika.adoc
+++ b/solr/solr-ref-guide/src/uploading-data-with-solr-cell-using-apache-tika.adoc
@@ -172,7 +172,7 @@ Here is the order in which the Solr Cell framework, using the Extracting Request
[[UploadingDatawithSolrCellusingApacheTika-ConfiguringtheSolrExtractingRequestHandler]]
== Configuring the Solr ExtractingRequestHandler
-If you are not working with the supplied `sample_techproducts_configs `or` data_driven_schema_configs` <<config-sets.adoc#config-sets,config set>>, you must configure your own `solrconfig.xml` to know about the Jar's containing the `ExtractingRequestHandler` and its dependencies:
+If you are not working with the supplied `sample_techproducts_configs` or `_default` <<config-sets.adoc#config-sets,config set>>, you must configure your own `solrconfig.xml` to know about the Jar's containing the `ExtractingRequestHandler` and its dependencies:
[source,xml]
----
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/04054fc5/solr/solr-ref-guide/src/using-zookeeper-to-manage-configuration-files.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/using-zookeeper-to-manage-configuration-files.adoc b/solr/solr-ref-guide/src/using-zookeeper-to-manage-configuration-files.adoc
index a9e0d71..3166e1c 100644
--- a/solr/solr-ref-guide/src/using-zookeeper-to-manage-configuration-files.adoc
+++ b/solr/solr-ref-guide/src/using-zookeeper-to-manage-configuration-files.adoc
@@ -31,7 +31,7 @@ These files are uploaded in either of the following cases:
When you try SolrCloud for the first time using the `bin/solr -e cloud`, the related configset gets uploaded to ZooKeeper automatically and is linked with the newly created collection.
-The below command would start SolrCloud with the default collection name (gettingstarted) and default configset (data_driven_schema_configs) uploaded and linked to it.
+The below command would start SolrCloud with the default collection name (gettingstarted) and default configset (_default) uploaded and linked to it.
[source,bash]
----
@@ -42,10 +42,10 @@ You can also explicitly upload a configuration directory when creating a collect
[source,bash]
----
-bin/solr create -c mycollection -d data_driven_schema_configs
+bin/solr create -c mycollection -d _default
----
-The create command will upload a copy of the `data_driven_schema_configs` configuration directory to ZooKeeper under `/configs/mycollection`. Refer to the <<solr-control-script-reference.adoc#solr-control-script-reference,Solr Control Script Reference>> page for more details about the create command for creating collections.
+The create command will upload a copy of the `_default` configuration directory to ZooKeeper under `/configs/mycollection`. Refer to the <<solr-control-script-reference.adoc#solr-control-script-reference,Solr Control Script Reference>> page for more details about the create command for creating collections.
Once a configuration directory has been uploaded to ZooKeeper, you can update them using the <<solr-control-script-reference.adoc#solr-control-script-reference,Solr Control Script>>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/04054fc5/solr/solr-ref-guide/src/velocity-response-writer.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/velocity-response-writer.adoc b/solr/solr-ref-guide/src/velocity-response-writer.adoc
index 3b576f6..424a033 100644
--- a/solr/solr-ref-guide/src/velocity-response-writer.adoc
+++ b/solr/solr-ref-guide/src/velocity-response-writer.adoc
@@ -18,7 +18,7 @@
// specific language governing permissions and limitations
// under the License.
-The VelocityResponseWriter is an optional plugin available in the `contrib/velocity` directory. It powers the /browse user interfaces when using configurations such as "basic_configs", "techproducts", and "example/files".
+The VelocityResponseWriter is an optional plugin available in the `contrib/velocity` directory. It powers the /browse user interfaces when using configurations such as "_default", "techproducts", and "example/files".
Its JAR and dependencies must be added (via `<lib>` or solr/home lib inclusion), and must be registered in `solrconfig.xml` like this:
[2/2] lucene-solr:branch_7_0: SOLR-9526: Data driven schema now
indexes text field "foo" as both "foo" (text_general) and as "foo_str"
(string) to facilitate both search and faceting
Posted by ja...@apache.org.
SOLR-9526: Data driven schema now indexes text field "foo" as both "foo" (text_general) and as "foo_str" (string) to facilitate both search and faceting
(cherry picked from commit a60ec1b)
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/257883d6
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/257883d6
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/257883d6
Branch: refs/heads/branch_7_0
Commit: 257883d65c4cc4c366493a6d0cae908fbccaca8f
Parents: 04054fc
Author: Jan Høydahl <ja...@apache.org>
Authored: Thu Jul 6 15:56:51 2017 +0200
Committer: Jan Høydahl <ja...@apache.org>
Committed: Thu Jul 6 16:24:18 2017 +0200
----------------------------------------------------------------------
solr/CHANGES.txt | 14 +-
.../apache/solr/schema/ManagedIndexSchema.java | 7 +-
.../AddSchemaFieldsUpdateProcessorFactory.java | 173 +++++++++++++++++--
...chema-add-schema-fields-update-processor.xml | 1 +
...dd-schema-fields-update-processor-chains.xml | 74 ++++++++
.../configsets/_default/conf/managed-schema | 3 +
.../configsets/_default/conf/solrconfig.xml | 14 +-
...dSchemaFieldsUpdateProcessorFactoryTest.java | 79 +++++++++
.../configsets/_default/conf/managed-schema | 3 +
.../configsets/_default/conf/solrconfig.xml | 14 +-
solr/solr-ref-guide/src/getting-started.adoc | 2 +-
solr/solr-ref-guide/src/schemaless-mode.adoc | 105 ++++++-----
.../transforming-and-indexing-custom-json.adoc | 4 +-
13 files changed, 425 insertions(+), 68 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/257883d6/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 8b28d59..5a75c66 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -120,11 +120,15 @@ Upgrading from Solr 6.x
* The unused 'valType' option has been removed from ExternalFileField, if you have this in your schema you
can safely remove it. see SOLR-10929 for more details.
-* SOLR-10574: basic_configs and data_driven_schema_configs have now been merged into _default. It has data driven nature
+* Config sets basic_configs and data_driven_schema_configs have now been merged into _default. It has data driven nature
enabled by default, and can be turned off (after creating a collection) with:
curl http://host:8983/solr/mycollection/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
Please see SOLR-10574 for details.
+* The data driven config (now _default) for auto-creating fields earlier defaulted to "string" for text input.
+ Now the default is to use "text_general" and to add a copyField to the schema, copying to a "*_str" dynamic field,
+ with a cutoff at 256 characters. This enables full text search as well as faceting. See SOLR-9526 for more.
+
* SOLR-10123: The Analytics Component has been upgraded to support distributed collections, expressions over multivalued
fields, a new JSON request language, and more. DocValues are now required for any field used in the analytics expression
whereas previously docValues was not required. Please see SOLR-10123 for details.
@@ -191,10 +195,16 @@ New Features
* SOLR-10406: v2 API error messages list the URL request path as /solr/____v2/... when the original path was /v2/... (Cao Manh Dat, noble)
* SOLR-10574: New _default config set replacing basic_configs and data_driven_schema_configs.
- (Ishan Chattopadhyaya, noble, shalin, hossman, David Smiley, Jan Hoydahl, Alexandre Rafalovich)
+ (Ishan Chattopadhyaya, noble, shalin, hossman, David Smiley, janhoy, Alexandre Rafalovich)
* SOLR-10272: Use _default config set if no collection.configName is specified with CREATE (Ishan Chattopadhyaya)
+* SOLR-9526: Data driven schema now indexes text field "foo" as both "foo" (text_general) and as "foo_str" (string)
+ to facilitate both search and faceting. AddSchemaFieldsUpdateProcessor now has the ability to add a "copyField" to
+ the type mappings, with an optional maxChars limitation. You can also define one typeMappings as default.
+ This also solves issues SOLR-8495, SOLR-6966, and SOLR-7058
+ (janhoy, Steve Rowe, hossman, Alexandre Rafalovich, Shawn Heisey, Cao Manh Dat)
+
* SOLR-10123: Upgraded the Analytics Component to version 2.0 which now supports distributed collections, expressions over
multivalued fields, a new JSON request language, and more. DocValues are now required for any field used in the analytics
expression whereas previously docValues was not required. Please see SOLR-10123 for details. (Houston Putman)
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/257883d6/solr/core/src/java/org/apache/solr/schema/ManagedIndexSchema.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/schema/ManagedIndexSchema.java b/solr/core/src/java/org/apache/solr/schema/ManagedIndexSchema.java
index 3806dd3..078edfd 100644
--- a/solr/core/src/java/org/apache/solr/schema/ManagedIndexSchema.java
+++ b/solr/core/src/java/org/apache/solr/schema/ManagedIndexSchema.java
@@ -109,8 +109,11 @@ public final class ManagedIndexSchema extends IndexSchema {
}
- /** Persist the schema to local storage or to ZooKeeper */
- boolean persistManagedSchema(boolean createOnly) {
+ /**
+ * Persist the schema to local storage or to ZooKeeper
+ * @param createOnly set to false to allow update of existing schema
+ */
+ public boolean persistManagedSchema(boolean createOnly) {
if (loader instanceof ZkSolrResourceLoader) {
return persistManagedSchemaToZooKeeper(createOnly);
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/257883d6/solr/core/src/java/org/apache/solr/update/processor/AddSchemaFieldsUpdateProcessorFactory.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/update/processor/AddSchemaFieldsUpdateProcessorFactory.java b/solr/core/src/java/org/apache/solr/update/processor/AddSchemaFieldsUpdateProcessorFactory.java
index c4a122d..92774e6 100644
--- a/solr/core/src/java/org/apache/solr/update/processor/AddSchemaFieldsUpdateProcessorFactory.java
+++ b/solr/core/src/java/org/apache/solr/update/processor/AddSchemaFieldsUpdateProcessorFactory.java
@@ -26,6 +26,7 @@ import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
+import java.util.stream.Collectors;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.SolrInputDocument;
@@ -128,7 +129,11 @@ public class AddSchemaFieldsUpdateProcessorFactory extends UpdateRequestProcesso
private static final String VALUE_CLASS_PARAM = "valueClass";
private static final String FIELD_TYPE_PARAM = "fieldType";
private static final String DEFAULT_FIELD_TYPE_PARAM = "defaultFieldType";
-
+ private static final String COPY_FIELD_PARAM = "copyField";
+ private static final String DEST_PARAM = "dest";
+ private static final String MAX_CHARS_PARAM = "maxChars";
+ private static final String IS_DEFAULT_PARAM = "default";
+
private List<TypeMapping> typeMappings = Collections.emptyList();
private SelectorParams inclusions = new SelectorParams();
private Collection<SelectorParams> exclusions = new ArrayList<>();
@@ -152,16 +157,18 @@ public class AddSchemaFieldsUpdateProcessorFactory extends UpdateRequestProcesso
validateSelectorParams(exclusion);
}
Object defaultFieldTypeParam = args.remove(DEFAULT_FIELD_TYPE_PARAM);
- if (null == defaultFieldTypeParam) {
- throw new SolrException(SERVER_ERROR, "Missing required init param '" + DEFAULT_FIELD_TYPE_PARAM + "'");
- } else {
+ if (null != defaultFieldTypeParam) {
if ( ! (defaultFieldTypeParam instanceof CharSequence)) {
throw new SolrException(SERVER_ERROR, "Init param '" + DEFAULT_FIELD_TYPE_PARAM + "' must be a <str>");
}
+ defaultFieldType = defaultFieldTypeParam.toString();
}
- defaultFieldType = defaultFieldTypeParam.toString();
typeMappings = parseTypeMappings(args);
+ if (null == defaultFieldType && typeMappings.stream().noneMatch(TypeMapping::isDefault)) {
+ throw new SolrException(SERVER_ERROR, "Must specify either '" + DEFAULT_FIELD_TYPE_PARAM +
+ "' or declare one typeMapping as default.");
+ }
super.init(args);
}
@@ -207,8 +214,59 @@ public class AddSchemaFieldsUpdateProcessorFactory extends UpdateRequestProcesso
throw new SolrException(SERVER_ERROR,
"Each '" + TYPE_MAPPING_PARAM + "' <lst/> must contain at least one '" + VALUE_CLASS_PARAM + "' <str>");
}
- typeMappings.add(new TypeMapping(fieldType, valueClasses));
+ // isDefault (optional)
+ Boolean isDefault = false;
+ Object isDefaultObj = typeMappingNamedList.remove(IS_DEFAULT_PARAM);
+ if (null != isDefaultObj) {
+ if ( ! (isDefaultObj instanceof Boolean)) {
+ throw new SolrException(SERVER_ERROR, "'" + IS_DEFAULT_PARAM + "' init param must be a <bool>");
+ }
+ if (null != typeMappingNamedList.get(IS_DEFAULT_PARAM)) {
+ throw new SolrException(SERVER_ERROR,
+ "Each '" + COPY_FIELD_PARAM + "' <lst/> may contain only one '" + IS_DEFAULT_PARAM + "' <bool>");
+ }
+ isDefault = Boolean.parseBoolean(isDefaultObj.toString());
+ }
+
+ Collection<CopyFieldDef> copyFieldDefs = new ArrayList<>();
+ while (typeMappingNamedList.get(COPY_FIELD_PARAM) != null) {
+ Object copyFieldObj = typeMappingNamedList.remove(COPY_FIELD_PARAM);
+ if ( ! (copyFieldObj instanceof NamedList)) {
+ throw new SolrException(SERVER_ERROR, "'" + COPY_FIELD_PARAM + "' init param must be a <lst>");
+ }
+ NamedList copyFieldNamedList = (NamedList)copyFieldObj;
+ // dest
+ Object destObj = copyFieldNamedList.remove(DEST_PARAM);
+ if (null == destObj) {
+ throw new SolrException(SERVER_ERROR,
+ "Each '" + COPY_FIELD_PARAM + "' <lst/> must contain a '" + DEST_PARAM + "' <str>");
+ }
+ if ( ! (destObj instanceof CharSequence)) {
+ throw new SolrException(SERVER_ERROR, "'" + COPY_FIELD_PARAM + "' init param must be a <str>");
+ }
+ if (null != copyFieldNamedList.get(COPY_FIELD_PARAM)) {
+ throw new SolrException(SERVER_ERROR,
+ "Each '" + COPY_FIELD_PARAM + "' <lst/> may contain only one '" + COPY_FIELD_PARAM + "' <str>");
+ }
+ String dest = destObj.toString();
+ // maxChars (optional)
+ Integer maxChars = 0;
+ Object maxCharsObj = copyFieldNamedList.remove(MAX_CHARS_PARAM);
+ if (null != maxCharsObj) {
+ if ( ! (maxCharsObj instanceof Integer)) {
+ throw new SolrException(SERVER_ERROR, "'" + MAX_CHARS_PARAM + "' init param must be a <int>");
+ }
+ if (null != copyFieldNamedList.get(MAX_CHARS_PARAM)) {
+ throw new SolrException(SERVER_ERROR,
+ "Each '" + COPY_FIELD_PARAM + "' <lst/> may contain only one '" + MAX_CHARS_PARAM + "' <str>");
+ }
+ maxChars = Integer.parseInt(maxCharsObj.toString());
+ }
+ copyFieldDefs.add(new CopyFieldDef(dest, maxChars));
+ }
+ typeMappings.add(new TypeMapping(fieldType, valueClasses, isDefault, copyFieldDefs));
+
if (0 != typeMappingNamedList.size()) {
throw new SolrException(SERVER_ERROR,
"Unexpected '" + TYPE_MAPPING_PARAM + "' init sub-param(s): '" + typeMappingNamedList.toString() + "'");
@@ -233,11 +291,16 @@ public class AddSchemaFieldsUpdateProcessorFactory extends UpdateRequestProcesso
private static class TypeMapping {
public String fieldTypeName;
public Collection<String> valueClassNames;
+ public Collection<CopyFieldDef> copyFieldDefs;
public Set<Class<?>> valueClasses;
+ public Boolean isDefault;
- public TypeMapping(String fieldTypeName, Collection<String> valueClassNames) {
+ public TypeMapping(String fieldTypeName, Collection<String> valueClassNames, boolean isDefault,
+ Collection<CopyFieldDef> copyFieldDefs) {
this.fieldTypeName = fieldTypeName;
this.valueClassNames = valueClassNames;
+ this.isDefault = isDefault;
+ this.copyFieldDefs = copyFieldDefs;
// this.valueClasses population is delayed until the schema is available
}
@@ -257,6 +320,38 @@ public class AddSchemaFieldsUpdateProcessorFactory extends UpdateRequestProcesso
}
}
}
+
+ public boolean isDefault() {
+ return isDefault;
+ }
+ }
+
+ private static class CopyFieldDef {
+ private final String destGlob;
+ private final Integer maxChars;
+
+ public CopyFieldDef(String destGlob, Integer maxChars) {
+ this.destGlob = destGlob;
+ this.maxChars = maxChars;
+ if (destGlob.contains("*") && (!destGlob.startsWith("*") && !destGlob.endsWith("*"))) {
+ throw new SolrException(SERVER_ERROR, "dest '" + destGlob +
+ "' is invalid. Must either be a plain field name or start or end with '*'");
+ }
+ }
+
+ public Integer getMaxChars() {
+ return maxChars;
+ }
+
+ public String getDest(String srcFieldName) {
+ if (!destGlob.contains("*")) {
+ return destGlob;
+ } else if (destGlob.startsWith("*")) {
+ return srcFieldName + destGlob.substring(1);
+ } else {
+ return destGlob.substring(0,destGlob.length()-1) + srcFieldName;
+ }
+ }
}
private class AddSchemaFieldsUpdateProcessor extends UpdateRequestProcessor {
@@ -278,6 +373,8 @@ public class AddSchemaFieldsUpdateProcessorFactory extends UpdateRequestProcesso
IndexSchema oldSchema = cmd.getReq().getSchema();
for (;;) {
List<SchemaField> newFields = new ArrayList<>();
+ // Group copyField defs per field and then per maxChar, to adapt to IndexSchema API
+ Map<String,Map<Integer,List<CopyFieldDef>>> newCopyFields = new HashMap<>();
// build a selector each time through the loop b/c the schema we are
// processing may have changed
FieldNameSelector selector = buildSelector(oldSchema);
@@ -285,12 +382,20 @@ public class AddSchemaFieldsUpdateProcessorFactory extends UpdateRequestProcesso
getUnknownFields(selector, doc, unknownFields);
for (final Map.Entry<String,List<SolrInputField>> entry : unknownFields.entrySet()) {
String fieldName = entry.getKey();
- String fieldTypeName = mapValueClassesToFieldType(entry.getValue());
+ String fieldTypeName = defaultFieldType;
+ TypeMapping typeMapping = mapValueClassesToFieldType(entry.getValue());
+ if (typeMapping != null) {
+ fieldTypeName = typeMapping.fieldTypeName;
+ if (!typeMapping.copyFieldDefs.isEmpty()) {
+ newCopyFields.put(fieldName,
+ typeMapping.copyFieldDefs.stream().collect(Collectors.groupingBy(CopyFieldDef::getMaxChars)));
+ }
+ }
newFields.add(oldSchema.newField(fieldName, fieldTypeName, Collections.<String,Object>emptyMap()));
}
- if (newFields.isEmpty()) {
+ if (newFields.isEmpty() && newCopyFields.isEmpty()) {
// nothing to do - no fields will be added - exit from the retry loop
- log.debug("No fields to add to the schema.");
+ log.debug("No fields or copyFields to add to the schema.");
break;
} else if ( isImmutableConfigSet(core) ) {
final String message = "This ConfigSet is immutable.";
@@ -298,7 +403,7 @@ public class AddSchemaFieldsUpdateProcessorFactory extends UpdateRequestProcesso
}
if (log.isDebugEnabled()) {
StringBuilder builder = new StringBuilder();
- builder.append("Fields to be added to the schema: [");
+ builder.append("\nFields to be added to the schema: [");
boolean isFirst = true;
for (SchemaField field : newFields) {
builder.append(isFirst ? "" : ",");
@@ -307,20 +412,44 @@ public class AddSchemaFieldsUpdateProcessorFactory extends UpdateRequestProcesso
builder.append("{type=").append(field.getType().getTypeName()).append("}");
}
builder.append("]");
+ builder.append("\nCopyFields to be added to the schema: [");
+ isFirst = true;
+ for (String fieldName : newCopyFields.keySet()) {
+ builder.append(isFirst ? "" : ",");
+ isFirst = false;
+ builder.append("source=").append(fieldName).append("{");
+ for (List<CopyFieldDef> copyFieldDefList : newCopyFields.get(fieldName).values()) {
+ for (CopyFieldDef copyFieldDef : copyFieldDefList) {
+ builder.append("{dest=").append(copyFieldDef.getDest(fieldName));
+ builder.append(", maxChars=").append(copyFieldDef.getMaxChars()).append("}");
+ }
+ }
+ builder.append("}");
+ }
+ builder.append("]");
log.debug(builder.toString());
}
// Need to hold the lock during the entire attempt to ensure that
// the schema on the request is the latest
synchronized (oldSchema.getSchemaUpdateLock()) {
try {
- IndexSchema newSchema = oldSchema.addFields(newFields);
+ IndexSchema newSchema = oldSchema.addFields(newFields, Collections.emptyMap(), false);
+ // Add copyFields
+ for (String srcField : newCopyFields.keySet()) {
+ for (Integer maxChars : newCopyFields.get(srcField).keySet()) {
+ newSchema = newSchema.addCopyFields(srcField,
+ newCopyFields.get(srcField).get(maxChars).stream().map(f -> f.getDest(srcField)).collect(Collectors.toList()),
+ maxChars);
+ }
+ }
if (null != newSchema) {
+ ((ManagedIndexSchema)newSchema).persistManagedSchema(false);
core.setLatestSchema(newSchema);
cmd.getReq().updateSchemaToLatest();
- log.debug("Successfully added field(s) to the schema.");
+ log.debug("Successfully added field(s) and copyField(s) to the schema.");
break; // success - exit from the retry loop
} else {
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Failed to add fields.");
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Failed to add fields and/or copyFields.");
}
} catch (ManagedIndexSchema.FieldExistsException e) {
log.error("At least one field to be added already exists in the schema - retrying.");
@@ -360,11 +489,11 @@ public class AddSchemaFieldsUpdateProcessorFactory extends UpdateRequestProcesso
}
/**
- * Maps all given field values' classes to a field type using the configured type mapping rules.
+ * Maps all given field values' classes to a typeMapping object
*
* @param fields one or more (same-named) field values from one or more documents
*/
- private String mapValueClassesToFieldType(List<SolrInputField> fields) {
+ private TypeMapping mapValueClassesToFieldType(List<SolrInputField> fields) {
NEXT_TYPE_MAPPING: for (TypeMapping typeMapping : typeMappings) {
for (SolrInputField field : fields) {
NEXT_FIELD_VALUE: for (Object fieldValue : field.getValues()) {
@@ -379,10 +508,18 @@ public class AddSchemaFieldsUpdateProcessorFactory extends UpdateRequestProcesso
}
}
// Success! Each of this field's values is an instance of a mapped valueClass
- return typeMapping.fieldTypeName;
+ return typeMapping;
}
// At least one of this field's values is not an instance of any of the mapped valueClass-s
- return defaultFieldType;
+ // Return the typeMapping marked as default, if we have one, else return null to use fallback type
+ List<TypeMapping> defaultMappings = typeMappings.stream().filter(TypeMapping::isDefault).collect(Collectors.toList());
+ if (defaultMappings.size() > 1) {
+ throw new SolrException(SERVER_ERROR, "Only one typeMapping can be default");
+ } else if (defaultMappings.size() == 1) {
+ return defaultMappings.get(0);
+ } else {
+ return null;
+ }
}
private FieldNameSelector buildSelector(IndexSchema schema) {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/257883d6/solr/core/src/test-files/solr/collection1/conf/schema-add-schema-fields-update-processor.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/collection1/conf/schema-add-schema-fields-update-processor.xml b/solr/core/src/test-files/solr/collection1/conf/schema-add-schema-fields-update-processor.xml
index aa8d860..febc9ae 100644
--- a/solr/core/src/test-files/solr/collection1/conf/schema-add-schema-fields-update-processor.xml
+++ b/solr/core/src/test-files/solr/collection1/conf/schema-add-schema-fields-update-processor.xml
@@ -48,6 +48,7 @@
<field name="_version_" type="long" indexed="true" stored="true"/>
<field name="_root_" type="string" indexed="true" stored="true" multiValued="false"/>
+ <dynamicField name="*_str" type="string" stored="false" multiValued="true" docValues="true" useDocValuesAsStored="false"/>
<dynamicField name="*_t" type="text" indexed="true" stored="true"/>
<dynamicField name="*_ti" type="tint" indexed="true" stored="true"/>
<dynamicField name="*_tl" type="tlong" indexed="true" stored="true"/>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/257883d6/solr/core/src/test-files/solr/collection1/conf/solrconfig-add-schema-fields-update-processor-chains.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/collection1/conf/solrconfig-add-schema-fields-update-processor-chains.xml b/solr/core/src/test-files/solr/collection1/conf/solrconfig-add-schema-fields-update-processor-chains.xml
index f2e0f3e..b0b63df 100644
--- a/solr/core/src/test-files/solr/collection1/conf/solrconfig-add-schema-fields-update-processor-chains.xml
+++ b/solr/core/src/test-files/solr/collection1/conf/solrconfig-add-schema-fields-update-processor-chains.xml
@@ -69,6 +69,80 @@
<processor class="solr.AddSchemaFieldsUpdateProcessorFactory">
<str name="defaultFieldType">text</str>
<lst name="typeMapping">
+ <str name="valueClass">java.lang.String</str>
+ <str name="fieldType">text</str>
+ <lst name="copyField">
+ <str name="dest">*_str</str>
+ </lst>
+ </lst>
+ <lst name="typeMapping">
+ <str name="valueClass">java.lang.Boolean</str>
+ <str name="fieldType">boolean</str>
+ </lst>
+ <lst name="typeMapping">
+ <str name="valueClass">java.lang.Integer</str>
+ <str name="fieldType">pints</str>
+ </lst>
+ <lst name="typeMapping">
+ <str name="valueClass">java.lang.Float</str>
+ <str name="fieldType">pfloats</str>
+ </lst>
+ <lst name="typeMapping">
+ <str name="valueClass">java.util.Date</str>
+ <str name="fieldType">pdates</str>
+ </lst>
+ <lst name="typeMapping">
+ <str name="valueClass">java.lang.Long</str>
+ <str name="valueClass">java.lang.Integer</str>
+ <str name="fieldType">plongs</str>
+ </lst>
+ <lst name="typeMapping">
+ <str name="valueClass">java.lang.Number</str>
+ <str name="fieldType">pdoubles</str>
+ </lst>
+ </processor>
+ <processor class="solr.RunUpdateProcessorFactory" />
+ </updateRequestProcessorChain>
+
+ <updateRequestProcessorChain name="add-fields-maxchars">
+ <processor class="solr.AddSchemaFieldsUpdateProcessorFactory">
+ <str name="defaultFieldType">text</str>
+ <lst name="typeMapping">
+ <str name="valueClass">java.lang.String</str>
+ <str name="fieldType">text</str>
+ <lst name="copyField">
+ <str name="dest">*_str</str>
+ <int name="maxChars">10</int>
+ </lst>
+ <lst name="copyField">
+ <str name="dest">*_t</str>
+ <int name="maxChars">20</int>
+ </lst>
+ <lst name="copyField">
+ <str name="dest">*2_t</str>
+ <int name="maxChars">20</int>
+ </lst>
+ </lst>
+ </processor>
+ <processor class="solr.LogUpdateProcessorFactory" />
+ <processor class="solr.DistributedUpdateProcessorFactory" />
+ <processor class="solr.RunUpdateProcessorFactory" />
+ </updateRequestProcessorChain>
+
+ <!-- This chain has one of the typeMappings set as default=true, instead of falling back to the defaultFieldType -->
+ <updateRequestProcessorChain name="add-fields-default-mapping">
+ <processor class="solr.AddSchemaFieldsUpdateProcessorFactory">
+ <lst name="typeMapping">
+ <str name="valueClass">java.lang.String</str>
+ <str name="fieldType">text</str>
+ <lst name="copyField">
+ <str name="dest">*_str</str>
+ <int name="maxChars">10</int>
+ </lst>
+ <!-- Use as default mapping instead of defaultFieldType -->
+ <bool name="default">true</bool>
+ </lst>
+ <lst name="typeMapping">
<str name="valueClass">java.lang.Boolean</str>
<str name="fieldType">boolean</str>
</lst>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/257883d6/solr/core/src/test-files/solr/configsets/_default/conf/managed-schema
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/_default/conf/managed-schema b/solr/core/src/test-files/solr/configsets/_default/conf/managed-schema
index bdbbf8f..27c7bf3 100644
--- a/solr/core/src/test-files/solr/configsets/_default/conf/managed-schema
+++ b/solr/core/src/test-files/solr/configsets/_default/conf/managed-schema
@@ -137,6 +137,9 @@
<dynamicField name="*_d" type="double" indexed="true" stored="true"/>
<dynamicField name="*_ds" type="doubles" indexed="true" stored="true"/>
+ <!-- Type used for data-driven schema, to add a string copy for each text field -->
+ <dynamicField name="*_str" type="strings" stored="false" docValues="true" indexed="false" />
+
<dynamicField name="*_dt" type="date" indexed="true" stored="true"/>
<dynamicField name="*_dts" type="date" indexed="true" stored="true" multiValued="true"/>
<dynamicField name="*_p" type="location" indexed="true" stored="true"/>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/257883d6/solr/core/src/test-files/solr/configsets/_default/conf/solrconfig.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/_default/conf/solrconfig.xml b/solr/core/src/test-files/solr/configsets/_default/conf/solrconfig.xml
index f75303f..46a1b6a 100644
--- a/solr/core/src/test-files/solr/configsets/_default/conf/solrconfig.xml
+++ b/solr/core/src/test-files/solr/configsets/_default/conf/solrconfig.xml
@@ -1138,7 +1138,8 @@
Field type guessing update processors that will
attempt to parse string-typed field values as Booleans, Longs,
Doubles, or Dates, and then add schema fields with the guessed
- field types.
+ field types. Text content will be indexed as "text_general" as
+ well as a copy to a plain string version in *_str.
These require that the schema is both managed and mutable, by
declaring schemaFactory as ManagedIndexSchemaFactory, with
@@ -1177,7 +1178,16 @@
</arr>
</updateProcessor>
<updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields">
- <str name="defaultFieldType">strings</str>
+ <lst name="typeMapping">
+ <str name="valueClass">java.lang.String</str>
+ <str name="fieldType">text_general</str>
+ <lst name="copyField">
+ <str name="dest">*_str</str>
+ <int name="maxChars">256</int>
+ </lst>
+ <!-- Use as default mapping instead of defaultFieldType -->
+ <bool name="default">true</bool>
+ </lst>
<lst name="typeMapping">
<str name="valueClass">java.lang.Boolean</str>
<str name="fieldType">booleans</str>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/257883d6/solr/core/src/test/org/apache/solr/update/processor/AddSchemaFieldsUpdateProcessorFactoryTest.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/update/processor/AddSchemaFieldsUpdateProcessorFactoryTest.java b/solr/core/src/test/org/apache/solr/update/processor/AddSchemaFieldsUpdateProcessorFactoryTest.java
index 64d1abf..d1a3a53 100644
--- a/solr/core/src/test/org/apache/solr/update/processor/AddSchemaFieldsUpdateProcessorFactoryTest.java
+++ b/solr/core/src/test/org/apache/solr/update/processor/AddSchemaFieldsUpdateProcessorFactoryTest.java
@@ -17,10 +17,12 @@
package org.apache.solr.update.processor;
import java.io.File;
+import java.util.Collections;
import java.util.Date;
import org.apache.commons.io.FileUtils;
import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.ModifiableSolrParams;
import org.apache.solr.schema.IndexSchema;
import org.joda.time.DateTime;
import org.joda.time.format.DateTimeFormat;
@@ -115,6 +117,29 @@ public class AddSchemaFieldsUpdateProcessorFactoryTest extends UpdateProcessorTe
schema = h.getCore().getLatestSchema();
assertNotNull(schema.getFieldOrNull(fieldName));
assertEquals("text", schema.getFieldType(fieldName).getTypeName());
+ assertEquals(0, schema.getCopyFieldProperties(true, Collections.singleton(fieldName), null).size());
+ assertU(commit());
+ assertQ(req("id:4")
+ ,"//arr[@name='" + fieldName + "']/str[.='" + fieldValue1.toString() + "']"
+ ,"//arr[@name='" + fieldName + "']/str[.='" + fieldValue2.toString() + "']"
+ ,"//arr[@name='" + fieldName + "']/str[.='" + fieldValue3.toString() + "']"
+ );
+ }
+
+ public void testSingleFieldDefaultTypeMappingRoundTrip() throws Exception {
+ IndexSchema schema = h.getCore().getLatestSchema();
+ final String fieldName = "newfield4";
+ assertNull(schema.getFieldOrNull(fieldName));
+ Float fieldValue1 = -13258.0f;
+ Double fieldValue2 = 8.4828800808E10;
+ String fieldValue3 = "blah blah";
+ SolrInputDocument d = processAdd
+ ("add-fields-default-mapping", doc(f("id", "4"), f(fieldName, fieldValue1, fieldValue2, fieldValue3)));
+ assertNotNull(d);
+ schema = h.getCore().getLatestSchema();
+ assertNotNull(schema.getFieldOrNull(fieldName));
+ assertEquals("text", schema.getFieldType(fieldName).getTypeName());
+ assertEquals(1, schema.getCopyFieldProperties(true, Collections.singleton(fieldName), null).size());
assertU(commit());
assertQ(req("id:4")
,"//arr[@name='" + fieldName + "']/str[.='" + fieldValue1.toString() + "']"
@@ -209,6 +234,60 @@ public class AddSchemaFieldsUpdateProcessorFactoryTest extends UpdateProcessorTe
,"//arr[@name='" + fieldName3 + "']/str[.='" + field3String2 + "']"
,"//arr[@name='" + fieldName4 + "']/date[.='" + field4Value1String + "']");
}
+
+ public void testStringWithCopyField() throws Exception {
+ IndexSchema schema = h.getCore().getLatestSchema();
+ final String fieldName = "stringField";
+ final String strFieldName = fieldName+"_str";
+ assertNull(schema.getFieldOrNull(fieldName));
+ String content = "This is a text that should be copied to a string field but not be cutoff";
+ SolrInputDocument d = processAdd("add-fields", doc(f("id", "1"), f(fieldName, content)));
+ assertNotNull(d);
+ schema = h.getCore().getLatestSchema();
+ assertNotNull(schema.getFieldOrNull(fieldName));
+ assertNotNull(schema.getFieldOrNull(strFieldName));
+ assertEquals("text", schema.getFieldType(fieldName).getTypeName());
+ assertEquals(1, schema.getCopyFieldProperties(true, Collections.singleton(fieldName), Collections.singleton(strFieldName)).size());
+ }
+
+ public void testStringWithCopyFieldAndMaxChars() throws Exception {
+ IndexSchema schema = h.getCore().getLatestSchema();
+ final String fieldName = "stringField";
+ final String strFieldName = fieldName+"_str";
+ assertNull(schema.getFieldOrNull(fieldName));
+ String content = "This is a text that should be copied to a string field and cutoff at 10 characters";
+ SolrInputDocument d = processAdd("add-fields-maxchars", doc(f("id", "1"), f(fieldName, content)));
+ assertNotNull(d);
+ System.out.println("Document is "+d);
+ schema = h.getCore().getLatestSchema();
+ assertNotNull(schema.getFieldOrNull(fieldName));
+ assertNotNull(schema.getFieldOrNull(strFieldName));
+ assertEquals("text", schema.getFieldType(fieldName).getTypeName());
+ // We have three copyFields, one with maxChars 10 and two with maxChars 20
+ assertEquals(3, schema.getCopyFieldProperties(true, Collections.singleton(fieldName), null).size());
+ assertEquals("The configured maxChars cutoff does not exist on the copyField", 10,
+ schema.getCopyFieldProperties(true, Collections.singleton(fieldName), Collections.singleton(strFieldName))
+ .get(0).get("maxChars"));
+ assertEquals("The configured maxChars cutoff does not exist on the copyField", 20,
+ schema.getCopyFieldProperties(true, Collections.singleton(fieldName), Collections.singleton(fieldName+"_t"))
+ .get(0).get("maxChars"));
+ assertEquals("The configured maxChars cutoff does not exist on the copyField", 20,
+ schema.getCopyFieldProperties(true, Collections.singleton(fieldName), Collections.singleton(fieldName+"2_t"))
+ .get(0).get("maxChars"));
+ }
+
+ public void testCopyFieldByIndexing() throws Exception {
+ String content = "This is a text that should be copied to a string field and cutoff at 10 characters";
+ SolrInputDocument d = processAdd("add-fields-default-mapping", doc(f("id", "1"), f("mynewfield", content)));
+ assertU(commit());
+
+ ModifiableSolrParams params = new ModifiableSolrParams();
+ params.add("q", "*:*").add("facet", "true").add("facet.field", "mynewfield_str");
+ assertQ(req(params)
+ , "*[count(//doc)=1]"
+ ,"//lst[@name='mynewfield_str']/int[@name='This is a '][.='1']"
+ );
+ }
@After
private void deleteCoreAndTempSolrHomeDirectory() throws Exception {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/257883d6/solr/server/solr/configsets/_default/conf/managed-schema
----------------------------------------------------------------------
diff --git a/solr/server/solr/configsets/_default/conf/managed-schema b/solr/server/solr/configsets/_default/conf/managed-schema
index bdbbf8f..27c7bf3 100644
--- a/solr/server/solr/configsets/_default/conf/managed-schema
+++ b/solr/server/solr/configsets/_default/conf/managed-schema
@@ -137,6 +137,9 @@
<dynamicField name="*_d" type="double" indexed="true" stored="true"/>
<dynamicField name="*_ds" type="doubles" indexed="true" stored="true"/>
+ <!-- Type used for data-driven schema, to add a string copy for each text field -->
+ <dynamicField name="*_str" type="strings" stored="false" docValues="true" indexed="false" />
+
<dynamicField name="*_dt" type="date" indexed="true" stored="true"/>
<dynamicField name="*_dts" type="date" indexed="true" stored="true" multiValued="true"/>
<dynamicField name="*_p" type="location" indexed="true" stored="true"/>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/257883d6/solr/server/solr/configsets/_default/conf/solrconfig.xml
----------------------------------------------------------------------
diff --git a/solr/server/solr/configsets/_default/conf/solrconfig.xml b/solr/server/solr/configsets/_default/conf/solrconfig.xml
index f75303f..46a1b6a 100644
--- a/solr/server/solr/configsets/_default/conf/solrconfig.xml
+++ b/solr/server/solr/configsets/_default/conf/solrconfig.xml
@@ -1138,7 +1138,8 @@
Field type guessing update processors that will
attempt to parse string-typed field values as Booleans, Longs,
Doubles, or Dates, and then add schema fields with the guessed
- field types.
+ field types. Text content will be indexed as "text_general" as
+ well as a copy to a plain string version in *_str.
These require that the schema is both managed and mutable, by
declaring schemaFactory as ManagedIndexSchemaFactory, with
@@ -1177,7 +1178,16 @@
</arr>
</updateProcessor>
<updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields">
- <str name="defaultFieldType">strings</str>
+ <lst name="typeMapping">
+ <str name="valueClass">java.lang.String</str>
+ <str name="fieldType">text_general</str>
+ <lst name="copyField">
+ <str name="dest">*_str</str>
+ <int name="maxChars">256</int>
+ </lst>
+ <!-- Use as default mapping instead of defaultFieldType -->
+ <bool name="default">true</bool>
+ </lst>
<lst name="typeMapping">
<str name="valueClass">java.lang.Boolean</str>
<str name="fieldType">booleans</str>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/257883d6/solr/solr-ref-guide/src/getting-started.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/getting-started.adoc b/solr/solr-ref-guide/src/getting-started.adoc
index e82e691..733fa91 100644
--- a/solr/solr-ref-guide/src/getting-started.adoc
+++ b/solr/solr-ref-guide/src/getting-started.adoc
@@ -39,5 +39,5 @@ This section helps you get Solr up and running quickly, and introduces you to th
[TIP]
====
-Solr includes a Quick Start tutorial which will be helpful if you are just starting out with Solr. You can find it online at http://lucene.apache.org/solr/quickstart.html, or in your Solr installation at `$SOLR_INSTALL_DIR/docs/quickstart.html`.
+Solr includes a Quick Start tutorial which will be helpful if you are just starting out with Solr. You can find it online at http://lucene.apache.org/solr/quickstart.html.
====
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/257883d6/solr/solr-ref-guide/src/schemaless-mode.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/schemaless-mode.adoc b/solr/solr-ref-guide/src/schemaless-mode.adoc
index bdc1888..30e7d51 100644
--- a/solr/solr-ref-guide/src/schemaless-mode.adoc
+++ b/solr/solr-ref-guide/src/schemaless-mode.adoc
@@ -70,7 +70,7 @@ You can use the `/schema/fields` <<schema-api.adoc#schema-api,Schema API>> to co
[[SchemalessMode-ConfiguringSchemalessMode]]
== Configuring Schemaless Mode
-As described above, there are three configuration elements that need to be in place to use Solr in schemaless mode. In the default (`_default`) config set included with Solr these are already configured. If, however, you would like to implement schemaless on your own, you should make the following changes.
+As described above, there are three configuration elements that need to be in place to use Solr in schemaless mode. In the `_default` config set included with Solr these are already configured. If, however, you would like to implement schemaless on your own, you should make the following changes.
[[SchemalessMode-EnableManagedSchema]]
=== Enable Managed Schema
@@ -94,18 +94,16 @@ The UpdateRequestProcessorChain allows Solr to guess field types, and you can de
[source,xml]
----
-<updateRequestProcessorChain name="add-unknown-fields-to-the-schema">
- <!-- UUIDUpdateProcessorFactory will generate an id if none is present in the incoming document -->
- <processor class="solr.UUIDUpdateProcessorFactory" />
- <processor class="solr.RemoveBlankFieldUpdateProcessorFactory"/>
- <processor class="solr.FieldNameMutatingUpdateProcessorFactory">
+ <updateProcessor class="solr.UUIDUpdateProcessorFactory" name="uuid"/>
+ <updateProcessor class="solr.RemoveBlankFieldUpdateProcessorFactory" name="remove-blank"/>
+ <updateProcessor class="solr.FieldNameMutatingUpdateProcessorFactory" name="field-name-mutating">
<str name="pattern">[^\w-\.]</str>
<str name="replacement">_</str>
- </processor>
- <processor class="solr.ParseBooleanFieldUpdateProcessorFactory"/>
- <processor class="solr.ParseLongFieldUpdateProcessorFactory"/>
- <processor class="solr.ParseDoubleFieldUpdateProcessorFactory"/>
- <processor class="solr.ParseDateFieldUpdateProcessorFactory">
+ </updateProcessor>
+ <updateProcessor class="solr.ParseBooleanFieldUpdateProcessorFactory" name="parse-boolean"/>
+ <updateProcessor class="solr.ParseLongFieldUpdateProcessorFactory" name="parse-long"/>
+ <updateProcessor class="solr.ParseDoubleFieldUpdateProcessorFactory" name="parse-double"/>
+ <updateProcessor class="solr.ParseDateFieldUpdateProcessorFactory" name="parse-date">
<arr name="format">
<str>yyyy-MM-dd'T'HH:mm:ss.SSSZ</str>
<str>yyyy-MM-dd'T'HH:mm:ss,SSSZ</str>
@@ -125,9 +123,18 @@ The UpdateRequestProcessorChain allows Solr to guess field types, and you can de
<str>yyyy-MM-dd HH:mm</str>
<str>yyyy-MM-dd</str>
</arr>
- </processor>
- <processor class="solr.AddSchemaFieldsUpdateProcessorFactory">
- <str name="defaultFieldType">strings</str>
+ </updateProcessor>
+ <updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields">
+ <lst name="typeMapping">
+ <str name="valueClass">java.lang.String</str>
+ <str name="fieldType">text_general</str>
+ <lst name="copyField">
+ <str name="dest">*_str</str>
+ <int name="maxChars">256</int>
+ </lst>
+ <!-- Use as default mapping instead of defaultFieldType -->
+ <bool name="default">true</bool>
+ </lst>
<lst name="typeMapping">
<str name="valueClass">java.lang.Boolean</str>
<str name="fieldType">booleans</str>
@@ -145,11 +152,15 @@ The UpdateRequestProcessorChain allows Solr to guess field types, and you can de
<str name="valueClass">java.lang.Number</str>
<str name="fieldType">pdoubles</str>
</lst>
- </processor>
- <processor class="solr.LogUpdateProcessorFactory"/>
- <processor class="solr.DistributedUpdateProcessorFactory"/>
- <processor class="solr.RunUpdateProcessorFactory"/>
-</updateRequestProcessorChain>
+ </updateProcessor>
+
+ <!-- The update.autoCreateFields property can be turned to false to disable schemaless mode -->
+ <updateRequestProcessorChain name="add-unknown-fields-to-the-schema" default="${update.autoCreateFields:true}"
+ processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
+ <processor class="solr.LogUpdateProcessorFactory"/>
+ <processor class="solr.DistributedUpdateProcessorFactory"/>
+ <processor class="solr.RunUpdateProcessorFactory"/>
+ </updateRequestProcessorChain>
----
Javadocs for update processor factories mentioned above:
@@ -166,7 +177,7 @@ Javadocs for update processor factories mentioned above:
[[SchemalessMode-MaketheUpdateRequestProcessorChaintheDefaultfortheUpdateRequestHandler]]
=== Make the UpdateRequestProcessorChain the Default for the UpdateRequestHandler
-Once the UpdateRequestProcessorChain has been defined, you must instruct your UpdateRequestHandlers to use it when working with index updates (i.e., adding, removing, replacing documents). Here is an example using <<initparams-in-solrconfig.adoc#initparams-in-solrconfig,InitParams>> to set the defaults on all `/update` request handlers:
+Once the UpdateRequestProcessorChain has been defined, you must instruct your UpdateRequestHandlers to use it when working with index updates (i.e., adding, removing, replacing documents). There are two ways to do this. The update chain shown above has a `default=true` attribute which will use it for any update handler. An alternative, more explicit way is to use <<initparams-in-solrconfig.adoc#initparams-in-solrconfig,InitParams>> to set the defaults on all `/update` request handlers:
[source,xml]
----
@@ -185,9 +196,9 @@ After each of these changes have been made, Solr should be restarted (or, you ca
[[SchemalessMode-ExamplesofIndexedDocuments]]
== Examples of Indexed Documents
-Once the schemaless mode has been enabled (whether you configured it manually or are using `_default` ), documents that include fields that are not defined in your schema should be added to the index, and the new fields added to the schema.
+Once the schemaless mode has been enabled (whether you configured it manually or are using `_default`), documents that include fields that are not defined in your schema will be indexed, using the guessed field types which are automatically added to the schema.
-For example, adding a CSV document will cause its fields that are not in the schema to be added, with fieldTypes based on values:
+For example, adding a CSV document will cause unknown fields to be added, with fieldTypes based on values:
[source,bash]
----
@@ -212,37 +223,51 @@ The fields now in the schema (output from `curl \http://localhost:8983/solr/gett
{
"responseHeader":{
"status":0,
- "QTime":1},
+ "QTime":2},
"fields":[{
"name":"Album",
- "type":"strings"}, // Field value guessed as String -> strings fieldType
+ "type":"text_general"},
{
"name":"Artist",
- "type":"strings"}, // Field value guessed as String -> strings fieldType
+ "type":"text_general"},
{
"name":"FromDistributor",
- "type":"tlongs"}, // Field value guessed as Long -> tlongs fieldType
+ "type":"plongs"},
{
"name":"Rating",
- "type":"tdoubles"}, // Field value guessed as Double -> tdoubles fieldType
+ "type":"pdoubles"},
{
"name":"Released",
- "type":"tdates"}, // Field value guessed as Date -> tdates fieldType
+ "type":"pdates"},
{
"name":"Sold",
- "type":"tlongs"}, // Field value guessed as Long -> tlongs fieldType
+ "type":"plongs"},
{
- "name":"_text_",
-...
- },
+ "name":"_root_" ...}
{
- "name":"_version_",
-...
- },
+ "name":"_text_" ...}
{
- "name":"id",
-...
- }]}
+ "name":"_version_" ...}
+ {
+ "name":"id" ...}
+----
+
+In addition string versions of the text fields are indexed, using copyFields to a `*_str` dynamic field: (output from `curl \http://localhost:8983/solr/gettingstarted/schema/copyfields` ):
+
+[source,json]
+----
+{
+ "responseHeader":{
+ "status":0,
+ "QTime":0},
+ "copyFields":[{
+ "source":"Artist",
+ "dest":"Artist_str",
+ "maxChars":256},
+ {
+ "source":"Album",
+ "dest":"Album_str",
+ "maxChars":256}]}
----
.You Can Still Be Explicit
@@ -251,9 +276,11 @@ The fields now in the schema (output from `curl \http://localhost:8983/solr/gett
Even if you want to use schemaless mode for most fields, you can still use the <<schema-api.adoc#schema-api,Schema API>> to pre-emptively create some fields, with explicit types, before you index documents that use them.
Internally, the Schema API and the Schemaless Update Processors both use the same <<schema-factory-definition-in-solrconfig.adoc#schema-factory-definition-in-solrconfig,Managed Schema>> functionality.
+
+Also, if you do not need the `*_str` version of a text field, you can simply remove the `copyField` definition from the auto-generated schema and it will not be re-added since the original field is now defined.
====
-Once a field has been added to the schema, its field type is fixed. As a consequence, adding documents with field value(s) that conflict with the previously guessed field type will fail. For example, after adding the above document, the "```Sold```" field has the fieldType `tlongs`, but the document below has a non-integral decimal value in this field:
+Once a field has been added to the schema, its field type is fixed. As a consequence, adding documents with field value(s) that conflict with the previously guessed field type will fail. For example, after adding the above document, the "```Sold```" field has the fieldType `plongs`, but the document below has a non-integral decimal value in this field:
[source,bash]
----
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/257883d6/solr/solr-ref-guide/src/transforming-and-indexing-custom-json.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/transforming-and-indexing-custom-json.adoc b/solr/solr-ref-guide/src/transforming-and-indexing-custom-json.adoc
index 5b858e3..a3ea40e 100644
--- a/solr/solr-ref-guide/src/transforming-and-indexing-custom-json.adoc
+++ b/solr/solr-ref-guide/src/transforming-and-indexing-custom-json.adoc
@@ -153,7 +153,7 @@ In this example, we simply named the field paths (such as `/exams/test`). Solr w
[TIP]
====
-If you are working in <<schemaless-mode.adoc#schemaless-mode,Schemaless Mode>>, fields that don't exist will be created on the fly with Solr's best guess for the field type. Documents WILL get rejected if the fields do not exist in the schema before indexing. So, if you are NOT using schemaless mode, pre-create those fields.
+Documents WILL get rejected if the fields do not exist in the schema before indexing. So, if you are NOT using schemaless mode, pre-create those fields. If you are working in <<schemaless-mode.adoc#schemaless-mode,Schemaless Mode>>, fields that don't exist will be created on the fly with Solr's best guess for the field type.
====
@@ -336,7 +336,7 @@ With this example, the documents indexed would be, as follows:
== Tips for Custom JSON Indexing
1. Schemaless mode: This handles field creation automatically. The field guessing may not be exactly as you expect, but it works. The best thing to do is to setup a local server in schemaless mode, index a few sample docs and create those fields in your real setup with proper field types before indexing
-2. Pre-created Schema : Post your docs to the `/update/`json`/docs` endpoint with `echo=true`. This gives you the list of field names you need to create. Create the fields before you actually index
+2. Pre-created Schema : Post your docs to the `/update/json/docs` endpoint with `echo=true`. This gives you the list of field names you need to create. Create the fields before you actually index
3. No schema, only full-text search : All you need to do is to do full-text search on your JSON. Set the configuration as given in the Setting JSON Defaults section.
[[TransformingandIndexingCustomJSON-SettingJSONDefaults]]