You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ct...@apache.org on 2017/05/06 01:43:38 UTC
lucene-solr:jira/solr-10290: SOLR-10296: content conversion, letter D
Repository: lucene-solr
Updated Branches:
refs/heads/jira/solr-10290 790541615 -> 645908e29
SOLR-10296: content conversion, letter D
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/645908e2
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/645908e2
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/645908e2
Branch: refs/heads/jira/solr-10290
Commit: 645908e293298ccce31988a717dfb122c3bf3611
Parents: 7905416
Author: Cassandra Targett <ct...@apache.org>
Authored: Fri May 5 20:42:53 2017 -0500
Committer: Cassandra Targett <ct...@apache.org>
Committed: Fri May 5 20:43:30 2017 -0500
----------------------------------------------------------------------
...adir-and-directoryfactory-in-solrconfig.adoc | 9 +++--
solr/solr-ref-guide/src/dataimport-screen.adoc | 5 ++-
solr/solr-ref-guide/src/de-duplication.adoc | 24 ++++++------
.../src/defining-core-properties.adoc | 24 ++++++------
solr/solr-ref-guide/src/defining-fields.adoc | 6 +--
.../detecting-languages-during-indexing.adoc | 14 ++++---
.../src/distributed-requests.adoc | 33 +++++++---------
.../distributed-search-with-index-sharding.adoc | 40 ++++++++++----------
solr/solr-ref-guide/src/documents-screen.adoc | 23 ++++++-----
solr/solr-ref-guide/src/docvalues.adoc | 19 ++--------
solr/solr-ref-guide/src/dynamic-fields.adoc | 9 ++---
11 files changed, 97 insertions(+), 109 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/645908e2/solr/solr-ref-guide/src/datadir-and-directoryfactory-in-solrconfig.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/datadir-and-directoryfactory-in-solrconfig.adoc b/solr/solr-ref-guide/src/datadir-and-directoryfactory-in-solrconfig.adoc
index a4b5312..e2a847f 100644
--- a/solr/solr-ref-guide/src/datadir-and-directoryfactory-in-solrconfig.adoc
+++ b/solr/solr-ref-guide/src/datadir-and-directoryfactory-in-solrconfig.adoc
@@ -2,17 +2,18 @@
:page-shortname: datadir-and-directoryfactory-in-solrconfig
:page-permalink: datadir-and-directoryfactory-in-solrconfig.html
-[[DataDirandDirectoryFactoryinSolrConfig-SpecifyingaLocationforIndexDatawiththedataDirParameter]]
+Where and how Solr stores its indexes are configurable options.
+
== Specifying a Location for Index Data with the `dataDir` Parameter
-By default, Solr stores its index data in a directory called `/data` under the core's instanceDir. If you would like to specify a different directory for storing index data, you can configure dataDir in the core.properties file for the core, or use the `<dataDir>` parameter in the `solrconfig.xml` file. You can specify another directory either with a full pathname or a pathname relative to the instanceDir of the SolrCore. For example:
+By default, Solr stores its index data in a directory called `/data` under the core's instance directory (`instanceDir`). If you would like to specify a different directory for storing index data, you can configure the `dataDir` in the `core.properties` file for the core, or use the `<dataDir>` parameter in the `solrconfig.xml` file. You can specify another directory either with an absolute path or a pathname relative to the instanceDir of the SolrCore. For example:
[source,xml]
----
<dataDir>/solr/data/${solr.core.name}</dataDir>
----
-$\{solr.core.name} will cause the name current core to be substituted, which results in each core's data being kept in a separate subdirectory.
+The `${solr.core.name}` substitution will cause the name of the current core to be substituted, which results in each core's data being kept in a separate subdirectory.
If you are using replication to replicate the Solr index (as described in <<legacy-scaling-and-distribution.adoc#legacy-scaling-and-distribution,Legacy Scaling and Distribution>>), then the `<dataDir>` directory should correspond to the index directory used in the replication configuration.
@@ -36,7 +37,7 @@ The {solr-javadocs}/solr-core/org/apache/solr/core/RAMDirectoryFactory.html[`sol
<directoryFactory class="org.apache.solr.core.RAMDirectoryFactory"/>
----
-[IMPORTANT]
+[NOTE]
====
If you are using Hadoop and would like to store your indexes in HDFS, you should use the {solr-javadocs}/solr-core/org/apache/solr/core/HdfsDirectoryFactory.html[`solr.HdfsDirectoryFactory`] instead of either of the above implementations. For more details, see the section <<running-solr-on-hdfs.adoc#running-solr-on-hdfs,Running Solr on HDFS>>.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/645908e2/solr/solr-ref-guide/src/dataimport-screen.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/dataimport-screen.adoc b/solr/solr-ref-guide/src/dataimport-screen.adoc
index 79c6f19..a90923e 100644
--- a/solr/solr-ref-guide/src/dataimport-screen.adoc
+++ b/solr/solr-ref-guide/src/dataimport-screen.adoc
@@ -4,7 +4,10 @@
The Dataimport screen shows the configuration of the DataImportHandler (DIH) and allows you start, and monitor the status of, import commands as defined by the options selected on the screen and defined in the configuration file.
+.The Dataimport Screen
image::images/dataimport-screen/dataimport.png[image,width=485,height=250]
-This screen also lets you adjust various options to control how the data is imported to Solr, and view the data import configuration file that controls the import. For more information about data importing with DIH, see the section on <<uploading-structured-data-store-data-with-the-data-import-handler.adoc#uploading-structured-data-store-data-with-the-data-import-handler,Uploading Structured Data Store Data with the Data Import Handler>>.
+This screen also lets you adjust various options to control how the data is imported to Solr, and view the data import configuration file that controls the import.
+
+For more information about data importing with DIH, see the section on <<uploading-structured-data-store-data-with-the-data-import-handler.adoc#uploading-structured-data-store-data-with-the-data-import-handler,Uploading Structured Data Store Data with the Data Import Handler>>.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/645908e2/solr/solr-ref-guide/src/de-duplication.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/de-duplication.adoc b/solr/solr-ref-guide/src/de-duplication.adoc
index 95cbeff..fee4408 100644
--- a/solr/solr-ref-guide/src/de-duplication.adoc
+++ b/solr/solr-ref-guide/src/de-duplication.adoc
@@ -2,23 +2,25 @@
:page-shortname: de-duplication
:page-permalink: de-duplication.html
-Preventing duplicate or near duplicate documents from entering an index or tagging documents with a signature/fingerprint for duplicate field collapsing can be efficiently achieved with a low collision or fuzzy hash algorithm. Solr natively supports de-duplication techniques of this type via the `<Signature>` class and allows for the easy addition of new hash/signature implementations. A Signature can be implemented several ways:
+If duplicate, or near-duplicate documents are a concern in your index, de-duplication may be worth implementing.
-[width="100%",cols="50%,50%",options="header",]
+Preventing duplicate or near duplicate documents from entering an index or tagging documents with a signature/fingerprint for duplicate field collapsing can be efficiently achieved with a low collision or fuzzy hash algorithm. Solr natively supports de-duplication techniques of this type via the `Signature` class and allows for the easy addition of new hash/signature implementations. A Signature can be implemented several ways:
+
+[width="100%",options="header",]
|===
|Method |Description
-|MD5Signature |128 bit hash used for exact duplicate detection.
-|Lookup3Signature |64 bit hash used for exact duplicate detection, much faster than MD5 and smaller to index
-|http://wiki.apache.org/solr/TextProfileSignature[TextProfileSignature] |Fuzzy hashing implementation from nutch for near duplicate detection. It's tunable but works best on longer text.
+|MD5Signature |128-bit hash used for exact duplicate detection.
+|Lookup3Signature |64-bit hash used for exact duplicate detection. This is much faster than MD5 and smaller to index.
+|http://wiki.apache.org/solr/TextProfileSignature[TextProfileSignature] |Fuzzy hashing implementation from Apache Nutch for near duplicate detection. It's tunable but works best on longer text.
|===
Other, more sophisticated algorithms for fuzzy/near hashing can be added later.
[IMPORTANT]
====
+Adding in the de-duplication process will change the `allowDups` setting so that it applies to an update term (with `signatureField` in this case) rather than the unique field Term.
-Adding in the de-duplication process will change the `allowDups` setting so that it applies to an update Term (with `signatureField` in this case) rather than the unique field Term. Of course the `signatureField` could be the unique field, but generally you want the unique field to be unique. When a document is added, a signature will automatically be generated and attached to the document in the specified `signatureField`.
-
+Of course the `signatureField` could be the unique field, but generally you want the unique field to be unique. When a document is added, a signature will automatically be generated and attached to the document in the specified `signatureField`.
====
[[De-Duplication-ConfigurationOptions]]
@@ -50,7 +52,7 @@ The `SignatureUpdateProcessorFactory` takes several properties:
// TODO: This table has cells that won't work with PDF: https://github.com/ctargett/refguide-asciidoc-poc/issues/13
-[width="100%",cols="34%,33%,33%",options="header",]
+[width="100%",options="header",]
|===
|Parameter |Default |Description
|signatureClass |`org.apache.solr.update.processor.Lookup3Signature` a|
@@ -69,7 +71,7 @@ A Signature implementation for generating a signature hash. The full classpath o
[[De-Duplication-Inschema.xml]]
=== In `schema.xml`
-If you are using a separate field for storing the signature you must have it indexed:
+If you are using a separate field for storing the signature, you must have it indexed:
[source,xml]
----
@@ -88,11 +90,9 @@ Be sure to change your update handlers to use the defined chain, as below:
</requestHandler>
----
-(This example assumes you have other sections of your request handler defined.)
+This example assumes you have other sections of your request handler defined.
[TIP]
====
-
The update processor can also be specified per request with a parameter of `update.chain=dedupe`.
-
====
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/645908e2/solr/solr-ref-guide/src/defining-core-properties.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/defining-core-properties.adoc b/solr/solr-ref-guide/src/defining-core-properties.adoc
index 1cd46c9..bdb7ba0 100644
--- a/solr/solr-ref-guide/src/defining-core-properties.adoc
+++ b/solr/solr-ref-guide/src/defining-core-properties.adoc
@@ -2,11 +2,13 @@
:page-shortname: defining-core-properties
:page-permalink: defining-core-properties.html
-Core discovery means that creating a core is as simple as a `core.properties` file located on disk. The `core.properties` file is a simple Java Properties file where each line is just a key=value pair, e.g., `name=core1`. Notice that no quotes are required.
+Core discovery means that creating a core is as simple as a `core.properties` file located on disk.
-A minimal `core.properties` file looks like this (however, it can also be empty, see information on placement of `core.properties` below):
+The `core.properties` file is a simple Java Properties file where each line is just a key=value pair, e.g., `name=core1`. Notice that no quotes are required.
-[source,text]
+A minimal `core.properties` file looks like the example below. However, it can also be empty, see information on placement of `core.properties` below.
+
+[source]
----
name=my_core_name
----
@@ -16,7 +18,7 @@ name=my_core_name
Solr cores are configured by placing a file named `core.properties` in a sub-directory under `solr.home`. There are no a-priori limits to the depth of the tree, nor are there limits to the number of cores that can be defined. Cores may be anywhere in the tree with the exception that cores may _not_ be defined under an existing core. That is, the following is not allowed:
-[source,text]
+[source]
----
./cores/core1/core.properties
./cores/core1/coremore/core5/core.properties
@@ -26,7 +28,7 @@ In this example, the enumeration will stop at "core1".
The following is legal:
-[source,text]
+[source]
----
./cores/somecores/core1/core.properties
./cores/somecores/core2/core.properties
@@ -36,13 +38,11 @@ The following is legal:
It is possible to segment Solr into multiple cores, each with its own configuration and indices. Cores may be dedicated to a single application or to very different ones, but all are administered through a common administration interface. You can create new Solr cores on the fly, shutdown cores, even replace one running core with another, all without ever stopping or restarting Solr.
-Your `core.properties` file can be empty if necessary. Suppose `core.properties` is located in `./cores/core1 ` (relative to `solr_home`) but is empty. In that case, the core name is assumed to be "core1". The instanceDir will be the folder containing `core.properties` (i.e., `./cores/core1`). The dataDir will be `../cores/core1/data`, etc.
+Your `core.properties` file can be empty if necessary. Suppose `core.properties` is located in `./cores/core1` (relative to `solr_home`) but is empty. In this case, the core name is assumed to be "core1". The instanceDir will be the folder containing `core.properties` (i.e., `./cores/core1`). The dataDir will be `../cores/core1/data`, etc.
[NOTE]
====
-
You can run Solr without configuring any cores.
-
====
[[Definingcore.properties-Definingcore.propertiesFiles]]
@@ -52,11 +52,11 @@ You can run Solr without configuring any cores.
The minimal `core.properties` file is an empty file, in which case all of the properties are defaulted appropriately.
-Java properties files allow the hash ("#") or bang ("!") characters to specify comment-to-end-of-line.
+Java properties files allow the hash (`#`) or bang (`!`) characters to specify comment-to-end-of-line.
This table defines the recognized properties:
-[width="100%",cols="50%,50%",options="header",]
+[width="100%",options="header",]
|===
|Property |Description
|`name` |The name of the SolrCore. You'll use this name to reference the SolrCore when running commands with the CoreAdminHandler.
@@ -65,8 +65,8 @@ This table defines the recognized properties:
|`dataDir` |The core's data directory (where indexes are stored) as either an absolute pathname, or a path relative to the value of `instanceDir`. This is `data` by default.
|`configSet` |The name of a defined configset, if desired, to use to configure the core (see the <<config-sets.adoc#config-sets,Config Sets>> for more details).
|`properties` |The name of the properties file for this core. The value can be an absolute pathname or a path relative to the value of `instanceDir`.
-|`transient` |If **true**, the core can be unloaded if Solr reaches the `transientCacheSize`. The default if not specified is **false**. Cores are unloaded in order of least recently used first. _Setting to *true* is not recommended in SolrCloud mode._
-|`loadOnStartup` |If **true**, the default if it is not specified, the core will loaded when Solr starts. _Setting to *false* is not recommended in SolrCloud mode._
+|`transient` |If *true*, the core can be unloaded if Solr reaches the `transientCacheSize`. The default if not specified is *false*. Cores are unloaded in order of least recently used first. _Setting to *true* is not recommended in SolrCloud mode._
+|`loadOnStartup` |If *true*, the default if it is not specified, the core will loaded when Solr starts. _Setting to *false* is not recommended in SolrCloud mode._
|`coreNodeName` |Used only in SolrCloud, this is a unique identifier for the node hosting this replica. By default a coreNodeName is generated automatically, but setting this attribute explicitly allows you to manually assign a new core to replace an existing replica. For example: when replacing a machine that has had a hardware failure by restoring from backups on a new machine with a new hostname or port..
|`ulogDir` |The absolute or relative directory for the update log for this core (SolrCloud).
|`shard` |The shard to assign this core to (SolrCloud).
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/645908e2/solr/solr-ref-guide/src/defining-fields.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/defining-fields.adoc b/solr/solr-ref-guide/src/defining-fields.adoc
index 3992a05..6de3aa5 100644
--- a/solr/solr-ref-guide/src/defining-fields.adoc
+++ b/solr/solr-ref-guide/src/defining-fields.adoc
@@ -20,15 +20,15 @@ The following example defines a field named `price` with a type named `float` an
[cols=",",options="header",]
|===
|Property |Description
-|name |The name of the field. Field names should consist of alphanumeric or underscore characters only and not start with a digit. This is not currently strictly enforced, but other field names will not have first class support from all components and back compatibility is not guaranteed. Names with both leading and trailing underscores (e.g. `_version_`) are reserved. Every field must have a `name`.
-|type |The name of the `fieldType` for this field. This will be found in the "```name```" attribute on the `fieldType` definition. Every field must have a `type`.
+|name |The name of the field. Field names should consist of alphanumeric or underscore characters only and not start with a digit. This is not currently strictly enforced, but other field names will not have first class support from all components and back compatibility is not guaranteed. Names with both leading and trailing underscores (e.g., `_version_`) are reserved. Every field must have a `name`.
+|type |The name of the `fieldType` for this field. This will be found in the `name` attribute on the `fieldType` definition. Every field must have a `type`.
|default |A default value that will be added automatically to any document that does not have a value in this field when it is indexed. If this property is not specified, there is no default.
|===
[[DefiningFields-OptionalFieldTypeOverrideProperties]]
=== Optional Field Type Override Properties
-Fields can have many of the same properties as field types. Properties from the table below which are specified on an individual field will override any explicit value for that property specified on the the `<fieldType/>` of the field, or any implicit default property value provided by the underlying `FieldType` implementation. The table below is reproduced from <<field-type-definitions-and-properties.adoc#field-type-definitions-and-properties,Field Type Definitions and Properties>>, which has more details:
+Fields can have many of the same properties as field types. Properties from the table below which are specified on an individual field will override any explicit value for that property specified on the the `fieldType` of the field, or any implicit default property value provided by the underlying `fieldType` implementation. The table below is reproduced from <<field-type-definitions-and-properties.adoc#field-type-definitions-and-properties,Field Type Definitions and Properties>>, which has more details:
[width="100%",options="header",]
|===
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/645908e2/solr/solr-ref-guide/src/detecting-languages-during-indexing.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/detecting-languages-during-indexing.adoc b/solr/solr-ref-guide/src/detecting-languages-during-indexing.adoc
index 7d4f483..19dce39 100644
--- a/solr/solr-ref-guide/src/detecting-languages-during-indexing.adoc
+++ b/solr/solr-ref-guide/src/detecting-languages-during-indexing.adoc
@@ -2,14 +2,18 @@
:page-shortname: detecting-languages-during-indexing
:page-permalink: detecting-languages-during-indexing.html
-Solr can identify languages and map text to language-specific fields during indexing using the `langid` UpdateRequestProcessor. Solr supports two implementations of this feature:
+Solr can identify languages and map text to language-specific fields during indexing using the `langid` UpdateRequestProcessor.
+
+Solr supports two implementations of this feature:
* Tika's language detection feature: http://tika.apache.org/0.10/detection.html
* LangDetect language detection: https://github.com/shuyo/language-detection
You can see a comparison between the two implementations here: http://blog.mikemccandless.com/2011/10/accuracy-and-performance-of-googles.html. In general, the LangDetect implementation supports more languages with higher performance.
-For specific information on each of these language identification implementations, including a list of supported languages for each, see the relevant project websites. For more information about the `langid` UpdateRequestProcessor, see the Solr wiki: http://wiki.apache.org/solr/LanguageDetection. For more information about language analysis in Solr, see <<language-analysis.adoc#language-analysis,Language Analysis>>.
+For specific information on each of these language identification implementations, including a list of supported languages for each, see the relevant project websites.
+
+For more information about language analysis in Solr, see <<language-analysis.adoc#language-analysis,Language Analysis>>.
[[DetectingLanguagesDuringIndexing-ConfiguringLanguageDetection]]
== Configuring Language Detection
@@ -21,7 +25,7 @@ You can configure the `langid` UpdateRequestProcessor in `solrconfig.xml`. Both
Here is an example of a minimal Tika `langid` configuration in `solrconfig.xml`:
-[source,java]
+[source,xml]
----
<processor class="org.apache.solr.update.processor.TikaLanguageIdentifierUpdateProcessorFactory">
<lst name="defaults">
@@ -36,7 +40,7 @@ Here is an example of a minimal Tika `langid` configuration in `solrconfig.xml`:
Here is an example of a minimal LangDetect `langid` configuration in `solrconfig.xml`:
-[source,java]
+[source,xml]
----
<processor class="org.apache.solr.update.processor.LangDetectLanguageIdentifierUpdateProcessorFactory">
<lst name="defaults">
@@ -51,7 +55,7 @@ Here is an example of a minimal LangDetect `langid` configuration in `solrconfig
As previously mentioned, both implementations of the `langid` UpdateRequestProcessor take the same parameters.
-[width="100%",cols="20%,20%,20%,20%,20%",options="header",]
+[width="100%",options="header",]
|===
|Parameter |Type |Default |Required |Description
|langid |Boolean |true |no |Enables and disables language detection.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/645908e2/solr/solr-ref-guide/src/distributed-requests.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/distributed-requests.adoc b/solr/solr-ref-guide/src/distributed-requests.adoc
index 1fa0326..9c51513 100644
--- a/solr/solr-ref-guide/src/distributed-requests.adoc
+++ b/solr/solr-ref-guide/src/distributed-requests.adoc
@@ -2,7 +2,9 @@
:page-shortname: distributed-requests
:page-permalink: distributed-requests.html
-When a Solr node receives a search request, that request is routed behind the scenes to a replica of a shard that is part of the collection being searched. The chosen replica acts as an aggregator: it creates internal requests to randomly chosen replicas of every shard in the collection, coordinates the responses, issues any subsequent internal requests as needed (for example, to refine facets values, or request additional stored fields), and constructs the final response for the client.
+When a Solr node receives a search request, the request is routed behind the scenes to a replica of a shard that is part of the collection being searched.
+
+The chosen replica acts as an aggregator: it creates internal requests to randomly chosen replicas of every shard in the collection, coordinates the responses, issues any subsequent internal requests as needed (for example, to refine facets values, or request additional stored fields), and constructs the final response for the client.
[[DistributedRequests-LimitingWhichShardsareQueried]]
== Limiting Which Shards are Queried
@@ -11,44 +13,44 @@ While one of the advantages of using SolrCloud is the ability to query very larg
Querying all shards for a collection should look familiar; it's as though SolrCloud didn't even come into play:
-[source,java]
+[source]
----
http://localhost:8983/solr/gettingstarted/select?q=*:*
----
If, on the other hand, you wanted to search just one shard, you can specify that shard by its logical ID, as in:
-[source,java]
+[source]
----
http://localhost:8983/solr/gettingstarted/select?q=*:*&shards=shard1
----
If you want to search a group of shard Ids, you can specify them together:
-[source,java]
+[source]
----
http://localhost:8983/solr/gettingstarted/select?q=*:*&shards=shard1,shard2
----
In both of the above examples, the shard Id(s) will be used to pick a random replica of that shard.
-Alternatively, you can specify the explict replicas you wish to use in place of a shard Ids:
+Alternatively, you can specify the explicit replicas you wish to use in place of a shard Ids:
-[source,java]
+[source]
----
http://localhost:8983/solr/gettingstarted/select?q=*:*&shards=localhost:7574/solr/gettingstarted,localhost:8983/solr/gettingstarted
----
Or you can specify a list of replicas to choose from for a single shard (for load balancing purposes) by using the pipe symbol (|):
-[source,java]
+[source]
----
http://localhost:8983/solr/gettingstarted/select?q=*:*&shards=localhost:7574/solr/gettingstarted|localhost:7500/solr/gettingstarted
----
And of course, you can specify a list of shards (seperated by commas) each defined by a list of replicas (seperated by pipes). In this example, 2 shards are queried, the first being a random replica from shard1, the second being a random replica from the explicit pipe delimited list:
-[source,java]
+[source]
----
http://localhost:8983/solr/gettingstarted/select?q=*:*&shards=shard1,localhost:7574/solr/gettingstarted|localhost:7500/solr/gettingstarted
----
@@ -58,7 +60,7 @@ http://localhost:8983/solr/gettingstarted/select?q=*:*&shards=shard1,localhost:7
You can directly configure aspects of the concurrency and thread-pooling used within distributed search in Solr. This allows for finer grained control and you can tune it to target your own specific requirements. The default configuration favors throughput over latency.
-To configure the standard handler, provide a configuration like this in the solrconfig.xml:
+To configure the standard handler, provide a configuration like this in `solrconfig.xml`:
[source,xml]
----
@@ -73,20 +75,13 @@ To configure the standard handler, provide a configuration like this in the solr
The parameters that can be specified are as follows:
-// TODO: This table has cells that won't work with PDF: https://github.com/ctargett/refguide-asciidoc-poc/issues/13
-
-[width="100%",cols="34%,33%,33%",options="header",]
+[width="100%",options="header",]
|===
|Parameter |Default |Explanation
|`socketTimeout` |0 (use OS default) |The amount of time in ms that a socket is allowed to wait.
|`connTimeout` |0 (use OS default) |The amount of time in ms that is accepted for binding / connecting a socket
|`maxConnectionsPerHost` |20 |The maximum number of concurrent connections that is made to each individual shard in a distributed search.
-|`maxConnections` a|
-....
-10000
-....
-
- |The total maximum number of concurrent connections in distributed searches.
+|`maxConnections` |`10000` |The total maximum number of concurrent connections in distributed searches.
|`corePoolSize` |0 |The retained lowest limit on the number of threads used in coordinating distributed search.
|`maximumPoolSize` |Integer.MAX_VALUE |The maximum number of threads used for coordinating distributed search.
|`maxThreadIdleTime` |5 seconds |The amount of time to wait for before threads are scaled back in response to a reduction in load.
@@ -94,8 +89,6 @@ The parameters that can be specified are as follows:
|`fairnessPolicy` |false |Chooses the JVM specifics dealing with fair policy queuing, if enabled distributed searches will be handled in a First in First out fashion at a cost to throughput. If disabled throughput will be favored over latency.
|===
-// OLD_CONFLUENCE_ID: DistributedRequests-ConfiguringstatsCache(DistributedIDF)
-
[[DistributedRequests-ConfiguringstatsCache_DistributedIDF_]]
== Configuring statsCache (Distributed IDF)
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/645908e2/solr/solr-ref-guide/src/distributed-search-with-index-sharding.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/distributed-search-with-index-sharding.adoc b/solr/solr-ref-guide/src/distributed-search-with-index-sharding.adoc
index e8e27ec..5343e12 100644
--- a/solr/solr-ref-guide/src/distributed-search-with-index-sharding.adoc
+++ b/solr/solr-ref-guide/src/distributed-search-with-index-sharding.adoc
@@ -2,6 +2,8 @@
:page-shortname: distributed-search-with-index-sharding
:page-permalink: distributed-search-with-index-sharding.html
+When using traditional index sharding, you will need to consider how to query your documents.
+
It is highly recommended that you use <<solrcloud.adoc#solrcloud,SolrCloud>> when needing to scale up or scale out. The setup described below is legacy and was used prior to the existence of SolrCloud. SolrCloud provides for a truly distributed set of features with support for things like automatic routing, leader election, optimistic concurrency and other sanity checks that are expected out of a distributed system.
Everything on this page is specific to legacy setup of distributed search. Users trying out SolrCloud should not follow any of the steps or information below.
@@ -20,19 +22,17 @@ In the legacy distributed mode, Solr does not calculate universal term/doc frequ
If a query request includes the `shards` parameter, the Solr server distributes the request across all the shards listed as arguments to the parameter. The `shards` parameter uses this syntax:
-`__host__` `:` `__port__` `/` `__base_url__` `[,` `__host__` `:` `__port__` `/` `__base_url__` `]*`
+`host:port/base_url,host:port/base_url*`
For example, the `shards` parameter below causes the search to be distributed across two Solr servers: *solr1* and **solr2**, both of which are running on port 8983:
-` http://localhost:8983/solr/core1/select?shards=solr1:8983/solr/core1,solr2:8983/solr/core1&indent=true&q=ipod+solr `
+`\http://localhost:8983/solr/core1/select?shards=solr1:8983/solr/core1,solr2:8983/solr/core1&indent=true&q=ipod+solr`
Rather than require users to include the shards parameter explicitly, it is usually preferred to configure this parameter as a default in the RequestHandler section of `solrconfig.xml`.
[IMPORTANT]
====
-
-Do not add the `shards` parameter to the standard requestHandler; otherwise, search queries may enter an infinite loop. Instead, define a new requestHandler that uses the `shards` parameter, and pass distributed search requests to that handler.
-
+Do not add the `shards` parameter to the standard request handler; doing so may cause search queries may enter an infinite loop. Instead, define a new request handler that uses the `shards` parameter, and pass distributed search requests to that handler.
====
With Legacy mode, only query requests are distributed. This includes requests to the SearchHandler (or any handler extending from `org.apache.solr.handler.component.SearchHandler`) using standard components that support distributed search.
@@ -65,16 +65,16 @@ Formerly a limitation was that TF/IDF relevancy computations only used shard-loc
[[DistributedSearchwithIndexSharding-AvoidingDistributedDeadlock]]
== Avoiding Distributed Deadlock
-Like in SolrCloud mode, inter-shard requests could lead to a distributed deadlock. It can be avoided by following the instructions <<distributed-requests.adoc#distributed-requests,here>>.
+Like in SolrCloud mode, inter-shard requests could lead to a distributed deadlock. It can be avoided by following the instructions in the section <<distributed-requests.adoc#distributed-requests,Distributed Requests>>.
[[DistributedSearchwithIndexSharding-TestingIndexShardingonTwoLocalServers]]
== Testing Index Sharding on Two Local Servers
-For simple functionality testing, it's easiest to just set up two local Solr servers on different ports. (In a production environment, of course, these servers would be deployed on separate machines.)
+For simple functional testing, it's easiest to just set up two local Solr servers on different ports. (In a production environment, of course, these servers would be deployed on separate machines.)
-1. Make two Solr home directories:
+. Make two Solr home directories and copy `solr.xml` into the new directories:
+
-[source,java]
+[source,bash]
----
mkdir example/nodes
mkdir example/nodes/node1
@@ -84,9 +84,9 @@ cp server/solr/solr.xml example/nodes/node1/.
mkdir example/nodes/node2
cp server/solr/solr.xml example/nodes/node2/.
----
-2. Start the two Solr instances
+. Start the two Solr instances
+
-[source,java]
+[source,bash]
----
# Start first node on port 8983
bin/solr start -s example/nodes/node1 -p 8983
@@ -94,25 +94,25 @@ bin/solr start -s example/nodes/node1 -p 8983
# Start second node on port 8984
bin/solr start -s example/nodes/node2 -p 8984
----
-3. Create a core on both the nodes with the sample_techproducts_configs.
+. Create a core on both the nodes with the sample_techproducts_configs.
+
-[source,java]
+[source,bash]
----
bin/solr create_core -c core1 -p 8983 -d sample_techproducts_configs
# Create a core on the Solr node running on port 8984
bin/solr create_core -c core1 -p 8984 -d sample_techproducts_configs
----
-4. In the third window, index an example document to each of the server:
+. In a third window, index an example document to each of the server:
+
-[source,java]
+[source,bash]
----
bin/post -c core1 example/exampledocs/monitor.xml -port 8983
bin/post -c core1 example/exampledocs/monitor2.xml -port 8984
----
-5. Search on the node on port 8983:
+. Search on the node on port 8983:
+
-[source,bash]
+[source]
----
curl http://localhost:8983/solr/core1/select?q=*:*&wt=xml&indent=true
----
@@ -121,7 +121,7 @@ This should bring back one document.
+
Search on the node on port 8984:
+
-[source,bash]
+[source]
----
curl http://localhost:8984/solr/core1/select?q=*:*&wt=xml&indent=true
----
@@ -130,14 +130,14 @@ This should also bring back a single document.
+
Now do a distributed search across both servers with your browser or `curl.` In the example below, an extra parameter 'fl' is passed to restrict the returned fields to id and name.
+
-[source,bash]
+[source]
----
curl http://localhost:8983/solr/core1/select?q=*:*&indent=true&shards=localhost:8983/solr/core1,localhost:8984/solr/core1&fl=id,name
----
+
This should contain both the documents as shown below:
+
-[source,java]
+[source,xml]
----
<response>
<lst name="responseHeader">
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/645908e2/solr/solr-ref-guide/src/documents-screen.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/documents-screen.adoc b/solr/solr-ref-guide/src/documents-screen.adoc
index 960e292..0753746 100644
--- a/solr/solr-ref-guide/src/documents-screen.adoc
+++ b/solr/solr-ref-guide/src/documents-screen.adoc
@@ -4,15 +4,24 @@
The Documents screen provides a simple form allowing you to execute various Solr indexing commands in a variety of formats directly from the browser.
+.The Documents Screen
image::images/documents-screen/documents_add_screen.png[image,height=400]
-
The screen allows you to:
* Copy documents in JSON, CSV or XML and submit them to the index
* Upload documents (in JSON, CSV or XML)
* Construct documents by selecting fields and field values
+
+[TIP]
+====
+There are other ways to load data, see also these sections:
+
+* <<uploading-data-with-index-handlers.adoc#uploading-data-with-index-handlers,Uploading Data with Index Handlers>>
+* <<uploading-data-with-solr-cell-using-apache-tika.adoc#uploading-data-with-solr-cell-using-apache-tika,Uploading Data with Solr Cell using Apache Tika>>
+====
+
The first step is to define the RequestHandler to use (aka, 'qt'). By default `/update` will be defined. To use Solr Cell, for example, change the request handler to `/update/extract`.
Then choose the Document Type to define the type of document to load. The remaining parameters will change depending on the document type selected.
@@ -22,7 +31,7 @@ Then choose the Document Type to define the type of document to load. The remain
When using the JSON document type, the functionality is similar to using a requestHandler on the command line. Instead of putting the documents in a curl command, they can instead be input into the Document entry box. The document structure should still be in proper JSON format.
-Then you can choose when documents should be added to the index (Commit Within), & whether existing documents should be overwritten with incoming documents with the same id (if this is not **true**, then the incoming documents will be dropped).
+Then you can choose when documents should be added to the index (Commit Within), & whether existing documents should be overwritten with incoming documents with the same id (if this is not *true*, then the incoming documents will be dropped).
This option will only add or overwrite documents to the index; for other update tasks, see the <<DocumentsScreen-SolrCommand,Solr Command>> option.
@@ -31,7 +40,7 @@ This option will only add or overwrite documents to the index; for other update
When using the CSV document type, the functionality is similar to using a requestHandler on the command line. Instead of putting the documents in a curl command, they can instead be input into the Document entry box. The document structure should still be in proper CSV format, with columns delimited and one row per document.
-Then you can choose when documents should be added to the index (Commit Within), and whether existing documents should be overwritten with incoming documents with the same id (if this is not **true**, then the incoming documents will be dropped).
+Then you can choose when documents should be added to the index (Commit Within), and whether existing documents should be overwritten with incoming documents with the same id (if this is not *true*, then the incoming documents will be dropped).
[[DocumentsScreen-DocumentBuilder]]
=== Document Builder
@@ -45,7 +54,7 @@ The File Upload option allows choosing a prepared file and uploading it. If usin
However, to use the ExtractingRequestHandler (aka Solr Cell), you can modify the Request-Handler to `/update/extract`. You must have this defined in your `solrconfig.xml` file, with your desired defaults. You should also update the `&literal.id` shown in the Extracting Req. Handler Params so the file chosen is given a unique id.
-Then you can choose when documents should be added to the index (Commit Within), and whether existing documents should be overwritten with incoming documents with the same id (if this is not **true**, then the incoming documents will be dropped).
+Then you can choose when documents should be added to the index (Commit Within), and whether existing documents should be overwritten with incoming documents with the same id (if this is not *true*, then the incoming documents will be dropped).
[[DocumentsScreen-SolrCommand]]
=== Solr Command
@@ -62,9 +71,3 @@ When using the XML document type, the functionality is similar to using a reques
Then you can choose when documents should be added to the index (Commit Within), and whether existing documents should be overwritten with incoming documents with the same id (if this is not **true**, then the incoming documents will be dropped).
This option will only add or overwrite documents to the index; for other update tasks, see the <<DocumentsScreen-SolrCommand,Solr Command>> option.
-
-[[DocumentsScreen-RelatedTopics]]
-== Related Topics
-
-* <<uploading-data-with-index-handlers.adoc#uploading-data-with-index-handlers,Uploading Data with Index Handlers>>
-* <<uploading-data-with-solr-cell-using-apache-tika.adoc#uploading-data-with-solr-cell-using-apache-tika,Uploading Data with Solr Cell using Apache Tika>>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/645908e2/solr/solr-ref-guide/src/docvalues.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/docvalues.adoc b/solr/solr-ref-guide/src/docvalues.adoc
index ab78f0c..d7ff9ad 100644
--- a/solr/solr-ref-guide/src/docvalues.adoc
+++ b/solr/solr-ref-guide/src/docvalues.adoc
@@ -4,12 +4,9 @@
DocValues are a way of recording field values internally that is more efficient for some purposes, such as sorting and faceting, than traditional indexing.
-// OLD_CONFLUENCE_ID: DocValues-WhyDocValues?
-
-[[DocValues-WhyDocValues_]]
== Why DocValues?
-The standard way that Solr builds the index is with an __inverted index__. This style builds a list of terms found in all the documents in the index and next to each term is a list of documents that the term appears in (as well as how many times the term appears in that document). This makes search very fast - since users search by terms, having a ready list of term-to-document values makes the query process faster.
+The standard way that Solr builds the index is with an _inverted index_. This style builds a list of terms found in all the documents in the index and next to each term is a list of documents that the term appears in (as well as how many times the term appears in that document). This makes search very fast - since users search by terms, having a ready list of term-to-document values makes the query process faster.
For other features that we now commonly associate with search, such as sorting, faceting, and highlighting, this approach is not very efficient. The faceting engine, for example, must look up each term that appears in each document that will make up the result set and pull the document IDs in order to build the facet list. In Solr, this is maintained in memory, and can be slow to load (depending on the number of documents, terms, etc.).
@@ -28,12 +25,8 @@ Enabling a field for docValues only requires adding `docValues="true"` to the fi
----
[IMPORTANT]
-====
-
If you have already indexed data into your Solr index, you will need to completely re-index your content after changing your field definitions in `schema.xml` in order to successfully use docValues.
-====
-
DocValues are only available for specific field types. The types chosen determine the underlying Lucene docValue type that will be used. The available Solr field types are:
* `StrField` and `UUIDField`.
@@ -59,19 +52,13 @@ There is an additional configuration option available, which is to modify the `d
Please note that the `docValuesFormat` option may change in future releases.
[NOTE]
-====
Lucene index back-compatibility is only supported for the default codec. If you choose to customize the `docValuesFormat` in your schema.xml, upgrading to a future version of Solr may require you to either switch back to the default codec and optimize your index to rewrite it into the default codec before upgrading, or re-build your entire index from scratch after upgrading.
-====
-[[DocValues-UsingDocValues]]
== Using DocValues
-// OLD_CONFLUENCE_ID: DocValues-Sorting,Faceting&Functions
-
-[[DocValues-Sorting_Faceting_Functions]]
=== Sorting, Faceting & Functions
-If `docValues="true"` for a field, then DocValues will automatically be used any time the field is used for <<common-query-parameters.adoc#CommonQueryParameters-ThesortParameter,sorting>>, <<faceting.adoc#faceting,faceting>> or <<function-queries.adoc#function-queries,Function Queries>>.
+If `docValues="true"` for a field, then DocValues will automatically be used any time the field is used for <<common-query-parameters.adoc#CommonQueryParameters-ThesortParameter,sorting>>, <<faceting.adoc#faceting,faceting>> or <<function-queries.adoc#function-queries,function queries>>.
[[DocValues-RetrievingDocValuesDuringSearch]]
=== Retrieving DocValues During Search
@@ -82,7 +69,7 @@ When `useDocValuesAsStored="false"`, non-stored DocValues fields can still be ex
In cases where the query is returning _only_ docValues fields performance may improve since returning stored fields requires disk reads and decompression whereas returning docValues fields in the fl list only requires memory access.
-When retrieving fields from their docValues form (using https://cwiki.apache.org/confluence/display/solr/Exporting+Result+Sets[/export handler], https://cwiki.apache.org/confluence/display/solr/Streaming+Expressions[streaming expressions] or if the field is requested in the `fl` parameter), two important differences between regular stored fields and docValues fields must be understood:
+When retrieving fields from their docValues form (using the <<exporting-result-sets.adoc#exporting-result-sets,/export handler>>, <<streaming-expressions.adoc#streaming-expressions,streaming expressions>> or if the field is requested in the `fl` parameter), two important differences between regular stored fields and docValues fields must be understood:
1. Order is _not_ preserved. For simply retrieving stored fields, the insertion order is the return order. For docValues, it is the _sorted_ order.
2. Multiple identical entries are collapsed into a single value. Thus if I insert values 4, 5, 2, 4, 1, my return will be 1, 2, 4, 5.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/645908e2/solr/solr-ref-guide/src/dynamic-fields.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/dynamic-fields.adoc b/solr/solr-ref-guide/src/dynamic-fields.adoc
index 3c79fa1..32888cc 100644
--- a/solr/solr-ref-guide/src/dynamic-fields.adoc
+++ b/solr/solr-ref-guide/src/dynamic-fields.adoc
@@ -2,7 +2,9 @@
:page-shortname: dynamic-fields
:page-permalink: dynamic-fields.html
-_Dynamic fields_ allow Solr to index fields that you did not explicitly define in your schema. This is useful if you discover you have forgotten to define one or more fields. Dynamic fields can make your application less brittle by providing some flexibility in the documents you can add to Solr.
+_Dynamic fields_ allow Solr to index fields that you did not explicitly define in your schema.
+
+This is useful if you discover you have forgotten to define one or more fields. Dynamic fields can make your application less brittle by providing some flexibility in the documents you can add to Solr.
A dynamic field is just like a regular field except it has a name with a wildcard in it. When you are indexing documents, a field that does not match any explicitly defined fields can be matched with a dynamic field.
@@ -16,8 +18,3 @@ Like regular fields, dynamic fields have a name, a field type, and options.
----
It is recommended that you include basic dynamic field mappings (like that shown above) in your `schema.xml`. The mappings can be very useful.
-
-[[DynamicFields-RelatedTopics]]
-== Related Topics
-
-* http://wiki.apache.org/solr/SchemaXml#Dynamic_fields[SchemaXML-Dynamic Fields]