You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ct...@apache.org on 2017/09/05 14:37:32 UTC
[1/2] lucene-solr:branch_7x: Ref Guide: update ref guide readme for
better explanation of build dependencies and targets
Repository: lucene-solr
Updated Branches:
refs/heads/branch_7x e0528f31d -> 1ecc0344e
Ref Guide: update ref guide readme for better explanation of build dependencies and targets
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/a4d63f34
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/a4d63f34
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/a4d63f34
Branch: refs/heads/branch_7x
Commit: a4d63f34a4e6cbcec5daec10769b149c6d6c0ae3
Parents: e0528f3
Author: Cassandra Targett <ct...@apache.org>
Authored: Fri Sep 1 12:22:49 2017 -0500
Committer: Cassandra Targett <ct...@apache.org>
Committed: Tue Sep 5 09:37:17 2017 -0500
----------------------------------------------------------------------
solr/solr-ref-guide/README.adoc | 32 +++++++++++++++++++++++---------
1 file changed, 23 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a4d63f34/solr/solr-ref-guide/README.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/README.adoc b/solr/solr-ref-guide/README.adoc
index a66d79d..b161396 100644
--- a/solr/solr-ref-guide/README.adoc
+++ b/solr/solr-ref-guide/README.adoc
@@ -18,26 +18,40 @@
This is the source for the Solr Reference Guide.
-Raw content is stored in Asciidoc (`.adoc`) formated files in the `src/` directory.
+Raw content is stored in Asciidoc (`.adoc`) formatted files in the `src/` directory.
+== Prerequisites for Building
These files are processed with AsciiDoctor in 2 different ways:
-* Via 'Jekyll' to build an HTML browsable version of the Ref Guide
-** Prerequisites: `Ruby` and the following gems must be installed:
-*** `jekyll`
-*** `jekyll-asciidoc`
-*** `pygments.rb`
-* Via `asciidoctor-ant` to build the officially released PDF version of the Ref Guide
-** Prerequisites: None (except for those required to use the Lucene/Solr build: Java, Ant)
+* Via Jekyll to build an HTML browsable version of the Ref Guide.
+** Prerequisites: `Ruby` (v2.1 or higher) and the following gems must be installed:
+*** `jekyll`: v3.5, not v4.x. Use `gem install --force --version 3.5.0 jekyll` to force install of Jekyll 3.5.0.
+*** `jekyll-asciidoc`: v2.1 or higher. Use `gem install jekyll-asciidoc` to install.
+*** `pygments.rb`: v1.1.2 or higher. Use `gem install pygments.rb` to install.
+* Via `asciidoctor-ant` to build the officially released PDF version of the Ref Guide.
+** Prerequisites: None beyond those required to use the main Lucene/Solr build: Java, and Ant.
+== Building the Guide
For details on building the ref guide, see `ant -p`.
+There are currently four available targets:
+
+* `ant default`: builds both the PDF and HTML versions of the Solr Ref Guide.
+* `ant build-site`: builds only the HTML version.
+* `ant build-pdf`: builds only the PDF version.
+* `ant clean`: removes the `../build/solr-ref-guide` directory.
+
+The output of all builds will be located in `../build/solr-ref-guide`.
+
+== Key Directories
Key directories to be aware of:
-* `src` - where all human edited `*.adoc` files realted to the Guide live, as well as various configuration, theme, and template files.
+* `src` - where all human edited `*.adoc` files related to the Guide live, as well as various configuration, theme, and template files.
* `tools` - custom Java code for parsing metadata in our `src/*.adoc` files to produce some `_data/` files for site & pdf navigation purposes.
* `../build/solr-ref-guide/content` - a copy of the `src` dir generated by ant where:
** `*.template` files are processed to replace ant properties with their runtime values
** some `../build/solr-ref-guide/content/_data` files are generated by our java tools based header attributes from each of the `*.adoc` files
* `../build/solr-ref-guide/html-site` - HTML generated version of the ref guide
* `../build/solr-ref-guide/apache-solr-ref-guide-X.Y.pdf` - PDF generated version of the ref guide
+
+See the additional documentation in `src/metadocs` for more information about how to edit files, build for releases, or modifying any Jekyll or PDF templates.
[2/2] lucene-solr:branch_7x: SOLR-9526: Update Ref Guide for
schemaless changes
Posted by ct...@apache.org.
SOLR-9526: Update Ref Guide for schemaless changes
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/1ecc0344
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/1ecc0344
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/1ecc0344
Branch: refs/heads/branch_7x
Commit: 1ecc0344ef45571bf8aaf84c8a37e8d18e17a0c2
Parents: a4d63f3
Author: Cassandra Targett <ct...@apache.org>
Authored: Tue Sep 5 09:35:45 2017 -0500
Committer: Cassandra Targett <ct...@apache.org>
Committed: Tue Sep 5 09:37:25 2017 -0500
----------------------------------------------------------------------
solr/solr-ref-guide/src/schemaless-mode.adoc | 62 ++++++++++++++++-------
1 file changed, 43 insertions(+), 19 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/1ecc0344/solr/solr-ref-guide/src/schemaless-mode.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/schemaless-mode.adoc b/solr/solr-ref-guide/src/schemaless-mode.adoc
index 825c294..f351bd6 100644
--- a/solr/solr-ref-guide/src/schemaless-mode.adoc
+++ b/solr/solr-ref-guide/src/schemaless-mode.adoc
@@ -22,7 +22,7 @@ Schemaless Mode is a set of Solr features that, when used together, allow users
These Solr features, all controlled via `solrconfig.xml`, are:
-. Managed schema: Schema modifications are made at runtime through Solr APIs, which requires the use of `schemaFactory` that supports these changes - see <<schema-factory-definition-in-solrconfig.adoc#schema-factory-definition-in-solrconfig,Schema Factory Definition in SolrConfig>> for more details.
+. Managed schema: Schema modifications are made at runtime through Solr APIs, which requires the use of a `schemaFactory` that supports these changes. See the section <<schema-factory-definition-in-solrconfig.adoc#schema-factory-definition-in-solrconfig,Schema Factory Definition in SolrConfig>> for more details.
. Field value class guessing: Previously unseen fields are run through a cascading set of value-based parsers, which guess the Java class of field values - parsers for Boolean, Integer, Long, Float, Double, and Date are currently available.
. Automatic schema field addition, based on field value class(es): Previously unseen fields are added to the schema, based on field value Java classes, which are mapped to schema field types - see <<solr-field-types.adoc#solr-field-types,Solr Field Types>>.
@@ -35,7 +35,7 @@ The three features of schemaless mode are pre-configured in the `_default` <<con
bin/solr start -e schemaless
----
-This will launch a Solr server, and automatically create a collection (named "```gettingstarted```") that contains only three fields in the initial schema: `id`, `\_version_`, and `\_text_`.
+This will launch a single Solr server, and automatically create a collection (named "```gettingstarted```") that contains only three fields in the initial schema: `id`, `\_version_`, and `\_text_`.
You can use the `/schema/fields` <<schema-api.adoc#schema-api,Schema API>> to confirm this: `curl \http://localhost:8983/solr/gettingstarted/schema/fields` will output:
@@ -84,19 +84,23 @@ You can configure the `ManagedIndexSchemaFactory` (and control the resource file
</schemaFactory>
----
-=== Define an UpdateRequestProcessorChain
+=== Enable Field Class Guessing
-The UpdateRequestProcessorChain allows Solr to guess field types, and you can define the default field type classes to use. To start, you should define it as follows (see the javadoc links below for update processor factory documentation):
+In Solr, an <<update-request-processors.adoc#update-request-processors,UpdateRequestProcessorChain>> defines a chain of plugins that are applied to documents before or while they are indexed.
+
+The field guessing aspect of Solr's schemaless mode uses a specially-defined UpdateRequestProcessorChain that allows Solr to guess field types. You can also define the default field type classes to use.
+
+To start, you should define it as follows (see the javadoc links below for update processor factory documentation):
[source,xml]
----
<updateProcessor class="solr.UUIDUpdateProcessorFactory" name="uuid"/>
<updateProcessor class="solr.RemoveBlankFieldUpdateProcessorFactory" name="remove-blank"/>
- <updateProcessor class="solr.FieldNameMutatingUpdateProcessorFactory" name="field-name-mutating">
+ <updateProcessor class="solr.FieldNameMutatingUpdateProcessorFactory" name="field-name-mutating"> <!--1-->
<str name="pattern">[^\w-\.]</str>
<str name="replacement">_</str>
</updateProcessor>
- <updateProcessor class="solr.ParseBooleanFieldUpdateProcessorFactory" name="parse-boolean"/>
+ <updateProcessor class="solr.ParseBooleanFieldUpdateProcessorFactory" name="parse-boolean"/> <!--2-->
<updateProcessor class="solr.ParseLongFieldUpdateProcessorFactory" name="parse-long"/>
<updateProcessor class="solr.ParseDoubleFieldUpdateProcessorFactory" name="parse-double"/>
<updateProcessor class="solr.ParseDateFieldUpdateProcessorFactory" name="parse-date">
@@ -120,11 +124,11 @@ The UpdateRequestProcessorChain allows Solr to guess field types, and you can de
<str>yyyy-MM-dd</str>
</arr>
</updateProcessor>
- <updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields">
+ <updateProcessor class="solr.AddSchemaFieldsUpdateProcessorFactory" name="add-schema-fields"> <!--3-->
<lst name="typeMapping">
- <str name="valueClass">java.lang.String</str>
+ <str name="valueClass">java.lang.String</str> <!--4-->
<str name="fieldType">text_general</str>
- <lst name="copyField">
+ <lst name="copyField"> <!--5-->
<str name="dest">*_str</str>
<int name="maxChars">256</int>
</lst>
@@ -140,7 +144,7 @@ The UpdateRequestProcessorChain allows Solr to guess field types, and you can de
<str name="fieldType">pdates</str>
</lst>
<lst name="typeMapping">
- <str name="valueClass">java.lang.Long</str>
+ <str name="valueClass">java.lang.Long</str> <!--6-->
<str name="valueClass">java.lang.Integer</str>
<str name="fieldType">plongs</str>
</lst>
@@ -152,14 +156,26 @@ The UpdateRequestProcessorChain allows Solr to guess field types, and you can de
<!-- The update.autoCreateFields property can be turned to false to disable schemaless mode -->
<updateRequestProcessorChain name="add-unknown-fields-to-the-schema" default="${update.autoCreateFields:true}"
- processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
+ processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields"> <!--7-->
<processor class="solr.LogUpdateProcessorFactory"/>
<processor class="solr.DistributedUpdateProcessorFactory"/>
<processor class="solr.RunUpdateProcessorFactory"/>
</updateRequestProcessorChain>
----
-Javadocs for update processor factories mentioned above:
+There are many things defined in this chain. Let's step through a few of them.
+
+<1> First, we're using the FieldNameMutatingUpdateProcessorFactory to lower-case all field names. Note that this and every following `<processor>` element include a `name`. These names will be used in the final chain definition at the end of this example.
+<2> Next we add several update request processors to parse different field types. Note the ParseDateFieldUpdateProcessorFactory includes a long list of possible date formations that would be parsed into valid Solr dates. If you have a custom date, you could add it to this list (see the link to the Javadocs below to get information on how).
+<3> Once the fields have been parsed, we define the field types that will be assigned to those fields. You can modify any of these that you would like to change.
+<4> In this definition, if the parsing step decides the incoming data in a field is a string, we will put this into a field in Solr with the field type `text_general`. This field type by default allows Solr to query on this field.
+<5> After we've added the `text_general` field, we have also defined a copy field rule that will copy all data from the new `text_general` field to a field with the same name suffixed with `_str`. This is done by Solr's dynamic fields feature. By defining the target of the copy field rule as a dynamic field in this way, you can control the field type used in your schema. The default selection allows Solr to facet, highlight, and sort on these fields.
+<6> This is another example of a mapping rule. In this case we define that when either of the `Long` or `Integer` field parsers identify a field, they should both map their fields to the `plongs` field type.
+<7> Finally, we add a chain definition that calls the list of plugins. These plugins are each called by the names we gave to them when we defined them. We can also add other processors to the chain, as shown here. Note we have also given the entire chain a `name` ("add-unknown-fields-to-the-schema"). We'll use this name in the next section to specify that our update request handler should use this chain definition.
+
+CAUTION: This chain definition will make a number of copy field rules for string fields to be created from corresponding text fields. If your data causes you to end up with a lot of copy field rules, indexing may be slowed down noticeably, and your index size will be larger. To control for these issues, it's recommended that you review the copy field rules that are created, and remove any which you do not need for faceting, sorting, highlighting, etc.
+
+If you're interested in more information about the classes used in this chain, here are links to the Javadocs for update processor factories mentioned above:
* {solr-javadocs}/solr-core/org/apache/solr/update/processor/UUIDUpdateProcessorFactory.html[UUIDUpdateProcessorFactory]
* {solr-javadocs}/solr-core/org/apache/solr/update/processor/RemoveBlankFieldUpdateProcessorFactory.html[RemoveBlankFieldUpdateProcessorFactory]
@@ -170,9 +186,13 @@ Javadocs for update processor factories mentioned above:
* {solr-javadocs}/solr-core/org/apache/solr/update/processor/ParseDateFieldUpdateProcessorFactory.html[ParseDateFieldUpdateProcessorFactory]
* {solr-javadocs}/solr-core/org/apache/solr/update/processor/AddSchemaFieldsUpdateProcessorFactory.html[AddSchemaFieldsUpdateProcessorFactory]
-=== Make the UpdateRequestProcessorChain the Default for the UpdateRequestHandler
+=== Set the Default UpdateRequestProcessorChain
+
+Once the UpdateRequestProcessorChain has been defined, you must instruct your UpdateRequestHandlers to use it when working with index updates (i.e., adding, removing, replacing documents).
+
+There are two ways to do this. The update chain shown above has a `default=true` attribute which will use it for any update handler.
-Once the UpdateRequestProcessorChain has been defined, you must instruct your UpdateRequestHandlers to use it when working with index updates (i.e., adding, removing, replacing documents). There are two ways to do this. The update chain shown above has a `default=true` attribute which will use it for any update handler. An alternative, more explicit way is to use <<initparams-in-solrconfig.adoc#initparams-in-solrconfig,InitParams>> to set the defaults on all `/update` request handlers:
+An alternative, more explicit way is to use <<initparams-in-solrconfig.adoc#initparams-in-solrconfig,InitParams>> to set the defaults on all `/update` request handlers:
[source,xml]
----
@@ -183,14 +203,18 @@ Once the UpdateRequestProcessorChain has been defined, you must instruct your Up
</initParams>
----
-[IMPORTANT]
-====
-After each of these changes have been made, Solr should be restarted (or, you can reload the cores to load the new `solrconfig.xml` definitions).
-====
+IMPORTANT: After all of these changes have been made, Solr should be restarted or the cores reloaded.
+
+=== Disabling Automatic Field Guessing
+
+Automatic field creation can be disabled with the `update.autoCreateFields` property. To do this, you can use the <<config-api.adoc#config-api,Config API>> with a command such as:
+
+[source,bash]
+curl http://host:8983/solr/mycollection/config -d '{"set-user-property": {"update.autoCreateFields":"false"}}'
== Examples of Indexed Documents
-Once the schemaless mode has been enabled (whether you configured it manually or are using `_default`), documents that include fields that are not defined in your schema will be indexed, using the guessed field types which are automatically added to the schema.
+Once the schemaless mode has been enabled (whether you configured it manually or are using the `_default` configset), documents that include fields that are not defined in your schema will be indexed, using the guessed field types which are automatically added to the schema.
For example, adding a CSV document will cause unknown fields to be added, with fieldTypes based on values: