You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ct...@apache.org on 2017/05/07 16:13:11 UTC
lucene-solr:jira/solr-10290: SOLR-10296: conversion, letter I
Repository: lucene-solr
Updated Branches:
refs/heads/jira/solr-10290 8916dd05a -> d77278df6
SOLR-10296: conversion, letter I
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/d77278df
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/d77278df
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/d77278df
Branch: refs/heads/jira/solr-10290
Commit: d77278df6bb7d0639c18d2cc6f8cee3ff9196c96
Parents: 8916dd0
Author: Cassandra Targett <ct...@apache.org>
Authored: Sun May 7 11:12:50 2017 -0500
Committer: Cassandra Targett <ct...@apache.org>
Committed: Sun May 7 11:12:50 2017 -0500
----------------------------------------------------------------------
solr/solr-ref-guide/src/index-replication.adoc | 65 +++++++++++---------
.../src/indexconfig-in-solrconfig.adoc | 34 ++++++----
.../src/indexing-and-basic-data-operations.adoc | 20 +++---
solr/solr-ref-guide/src/indexupgrader-tool.adoc | 4 +-
.../src/initparams-in-solrconfig.adoc | 10 +--
solr/solr-ref-guide/src/installing-solr.adoc | 12 ++--
...ntroduction-to-scaling-and-distribution.adoc | 12 ++--
.../src/introduction-to-solr-indexing.adoc | 8 +--
8 files changed, 89 insertions(+), 76 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/d77278df/solr/solr-ref-guide/src/index-replication.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/index-replication.adoc b/solr/solr-ref-guide/src/index-replication.adoc
index 7a15d8e..e04222a 100644
--- a/solr/solr-ref-guide/src/index-replication.adoc
+++ b/solr/solr-ref-guide/src/index-replication.adoc
@@ -6,11 +6,10 @@ Index Replication distributes complete copies of a master index to one or more s
The figure below shows a Solr configuration using index replication. The master server's index is replicated on the slaves.
+.A Solr index can be replicated across multiple slave servers, which then process requests.
image::images/index-replication/worddav2b7e14725d898b4104cdd9c502fc77cd.png[image,width=159,height=235]
-_A Solr index can be replicated across multiple slave servers, which then process requests._
-
[[IndexReplication-IndexReplicationinSolr]]
== Index Replication in Solr
@@ -26,9 +25,9 @@ Solr includes a Java implementation of index replication that works over HTTP:
.Replication In SolrCloud
[NOTE]
====
+Although there is no explicit concept of "master/slave" nodes in a <<solrcloud.adoc#solrcloud,SolrCloud>> cluster, the `ReplicationHandler` discussed on this page is still used by SolrCloud as needed to support "shard recovery" – but this is done in a peer to peer manner.
-Although there is no explicit concept of "master/slave" nodes in a <<solrcloud.adoc#solrcloud,SolrCloud>> cluster, the `ReplicationHandler` discussed on this page is still used by SolrCloud as needed to support "shard recovery" – but this is done in a peer to peer manner. When using SolrCloud, the `ReplicationHandler` must be available via the `/replication` path. Solr does this implicitly unless overridden explicitly in your `solrconfig.xml`, but If you wish to override the default behavior, make certain that you do not explicitly set any of the "master" or "slave" configuration options mentioned below, or they will interfere with normal SolrCloud operation.
-
+When using SolrCloud, the `ReplicationHandler` must be available via the `/replication` path. Solr does this implicitly unless overridden explicitly in your `solrconfig.xml`, but if you wish to override the default behavior, make certain that you do not explicitly set any of the "master" or "slave" configuration options mentioned below, or they will interfere with normal SolrCloud operation.
====
[[IndexReplication-ReplicationTerminology]]
@@ -36,7 +35,7 @@ Although there is no explicit concept of "master/slave" nodes in a <<solrcloud.a
The table below defines the key terms associated with Solr replication.
-[width="100%",cols="50%,50%",options="header",]
+[width="100%",options="header",]
|===
|Term |Definition
|Index |A Lucene index is a directory of files. These files make up the searchable and returnable data of a Solr Core.
@@ -56,14 +55,14 @@ The table below defines the key terms associated with Solr replication.
In addition to `ReplicationHandler` configuration options specific to the master/slave roles, there are a few special configuration options that are generally supported (even when using SolrCloud).
* `maxNumberOfBackups` an integer value dictating the maximum number of backups this node will keep on disk as it receives `backup` commands.
-* Similar to most other request handlers in Solr you may configure a set of "<<requesthandlers-and-searchcomponents-in-solrconfig.adoc#RequestHandlersandSearchComponentsinSolrConfig-SearchHandlers,defaults, invariants, and/or appends>>" parameters corresponding with any request parameters supported by the `ReplicationHandler` when <<IndexReplication-HTTPAPICommandsfortheReplicationHandler,processing commands>>.
+* Similar to most other request handlers in Solr you may configure a set of <<requesthandlers-and-searchcomponents-in-solrconfig.adoc#RequestHandlersandSearchComponentsinSolrConfig-SearchHandlers,defaults, invariants, and/or appends>> parameters corresponding with any request parameters supported by the `ReplicationHandler` when <<IndexReplication-HTTPAPICommandsfortheReplicationHandler,processing commands>>.
[[IndexReplication-ConfiguringtheReplicationRequestHandleronaMasterServer]]
=== Configuring the Replication RequestHandler on a Master Server
Before running a replication, you should set the following parameters on initialization of the handler:
-[width="100%",cols="50%,50%",options="header",]
+[width="100%",options="header",]
|===
|Name |Description
|replicateAfter |String specifying action after which replication should occur. Valid values are commit, optimize, or startup. There can be multiple values for this parameter. If you use "startup", you need to have a "commit" and/or "optimize" entry also if you want to trigger replication on future commits or optimizes.
@@ -75,7 +74,7 @@ Before running a replication, you should set the following parameters on initial
The example below shows a possible 'master' configuration for the `ReplicationHandler`, including a fixed number of backups and an invariant setting for the `maxWriteMBPerSec` request parameter to prevent slaves from saturating its network interface
-[source,java]
+[source,xml]
----
<requestHandler name="/replication" class="solr.ReplicationHandler">
<lst name="master">
@@ -96,7 +95,7 @@ The example below shows a possible 'master' configuration for the `ReplicationHa
In the configuration file on the master server, include a line like the following:
-[source,java]
+[source,xml]
----
<str name="confFiles">solrconfig_slave.xml:solrconfig.xml,x.xml,y.xml</str>
----
@@ -110,7 +109,7 @@ On the master server, the file name of the slave configuration file can be anyth
The code below shows how to configure a ReplicationHandler on a slave.
-[source,java]
+[source,xml]
----
<requestHandler name="/replication" class="solr.ReplicationHandler">
<lst name="slave">
@@ -119,7 +118,7 @@ The code below shows how to configure a ReplicationHandler on a slave.
possible to pass on this as a request param for the fetchindex command -->
<str name="masterUrl">http://remote_host:port/solr/core_name/replication</str>
- <!-- Interval in which the slave should poll master. Format is HH:mm:ss .
+ <!-- Interval in which the slave should poll master. Format is HH:mm:ss .
If this is absent slave does not poll automatically.
But a fetchindex can be triggered from the admin or the http API -->
@@ -165,7 +164,7 @@ A master may be able to serve only so many slaves without affecting performance.
Here is an example of a ReplicationHandler configuration for a repeater:
-[source,java]
+[source,xml]
----
<requestHandler name="/replication" class="solr.ReplicationHandler">
<lst name="master">
@@ -184,7 +183,7 @@ Here is an example of a ReplicationHandler configuration for a repeater:
When a commit or optimize operation is performed on the master, the RequestHandler reads the list of file names which are associated with each commit point. This relies on the `replicateAfter` parameter in the configuration to decide which types of events should trigger replication.
-[width="100%",cols="50%,50%",options="header",]
+[width="100%",options="header",]
|===
|Setting on the Master |Description
|commit |Triggers replication whenever a commit is performed on the master index.
@@ -194,7 +193,7 @@ When a commit or optimize operation is performed on the master, the RequestHandl
The replicateAfter parameter can accept multiple arguments. For example:
-[source,java]
+[source,xml]
----
<str name="replicateAfter">startup</str>
<str name="replicateAfter">commit</str>
@@ -204,7 +203,9 @@ The replicateAfter parameter can accept multiple arguments. For example:
[[IndexReplication-SlaveReplication]]
== Slave Replication
-The master is totally unaware of the slaves. The slave continuously keeps polling the master (depending on the `pollInterval` parameter) to check the current index version of the master. If the slave finds out that the master has a newer version of the index it initiates a replication process. The steps are as follows:
+The master is totally unaware of the slaves.
+
+The slave continuously keeps polling the master (depending on the `pollInterval` parameter) to check the current index version of the master. If the slave finds out that the master has a newer version of the index it initiates a replication process. The steps are as follows:
* The slave issues a `filelist` command to get the list of the files. This command returns the names of the files as well as some metadata (for example, size, a lastmodified timestamp, an alias if any).
* The slave checks with its own index if it has any of those files in the local index. It then runs the filecontent command to download the missing files. This uses a custom format (akin to the HTTP chunked encoding) to download the full content or a part of each file. If the connection breaks in between, the download resumes from the point it failed. At any point, the slave tries 5 times before giving up a replication altogether.
@@ -228,7 +229,11 @@ If a replication involved downloading of at least one configuration file, the Re
[[IndexReplication-ResolvingCorruptionIssuesonSlaveServers]]
=== Resolving Corruption Issues on Slave Servers
-If documents are added to the slave, then the slave is no longer in sync with its master. However, the slave will not undertake any action to put itself in sync, until the master has new index data. When a commit operation takes place on the master, the index version of the master becomes different from that of the slave. The slave then fetches the list of files and finds that some of the files present on the master are also present in the local index but with different sizes and timestamps. This means that the master and slave have incompatible indexes. To correct this problem, the slave then copies all the index files from master to a new index directory and asks the core to load the fresh index from the new directory.
+If documents are added to the slave, then the slave is no longer in sync with its master. However, the slave will not undertake any action to put itself in sync, until the master has new index data.
+
+When a commit operation takes place on the master, the index version of the master becomes different from that of the slave. The slave then fetches the list of files and finds that some of the files present on the master are also present in the local index but with different sizes and timestamps. This means that the master and slave have incompatible indexes.
+
+To correct this problem, the slave then copies all the index files from master to a new index directory and asks the core to load the fresh index from the new directory.
[[IndexReplication-HTTPAPICommandsfortheReplicationHandler]]
== HTTP API Commands for the ReplicationHandler
@@ -237,19 +242,19 @@ You can use the HTTP commands below to control the ReplicationHandler's operatio
// TODO: This table has cells that won't work with PDF: https://github.com/ctargett/refguide-asciidoc-poc/issues/13
-[width="100%",cols="50%,50%",options="header",]
+[width="100%",options="header",]
|===
|Command |Description
-|http://__master_host:port__/solr/__core_name__/replication?command=enablereplication |Enables replication on the master for all its slaves.
-|http://__master_host:port__/solr/__core_name__/replication?command=disablereplication |Disables replication on the master for all its slaves.
-|http://__host:port__/solr/__core_name__/replication?command=indexversion |Returns the version of the latest replicatable index on the specified master or slave.
-|http://__slave_host:port__/solr/__core_name__/replication?command=fetchindex |Forces the specified slave to fetch a copy of the index from its master. If you like, you can pass an extra attribute such as masterUrl or compression (or any other parameter which is specified in the `<lst name="slave">` tag) to do a one time replication from a master. This obviates the need for hard-coding the master in the slave.
-|http://__slave_host:port__/solr/__core_name__/replication?command=abortfetch |Aborts copying an index from a master to the specified slave.
-|http://__slave_host:port__/solr/__core_name__/replication?command=enablepoll |Enables the specified slave to poll for changes on the master.
-|http://__slave_host:port__/solr/__core_name__/replication?command=disablepoll |Disables the specified slave from polling for changes on the master.
-|http://__slave_host:port__/solr/__core_name__/replication?command=details |Retrieves configuration details and current status.
-|http://__host:port__/solr/__core_name__/replication?command=filelist&generation=<__generation-number__> |Retrieves a list of Lucene files present in the specified host's index. You can discover the generation number of the index by running the `indexversion` command.
-|http://__master_host:port__/solr/__core_name__/replication?command=backup a|
+|http://_master_host:port_/solr/_core_name_/replication?command=enablereplication |Enables replication on the master for all its slaves.
+|http://_master_host:port_/solr/_core_name_/replication?command=disablereplication |Disables replication on the master for all its slaves.
+|http://_host:port_/solr/_core_name_/replication?command=indexversion |Returns the version of the latest replicatable index on the specified master or slave.
+|http://_slave_host:port_/solr/_core_name_/replication?command=fetchindex |Forces the specified slave to fetch a copy of the index from its master. If you like, you can pass an extra attribute such as masterUrl or compression (or any other parameter which is specified in the `<lst name="slave">` tag) to do a one time replication from a master. This obviates the need for hard-coding the master in the slave.
+|http://_slave_host:port_/solr/_core_name_/replication?command=abortfetch |Aborts copying an index from a master to the specified slave.
+|http://_slave_host:port_/solr/_core_name_/replication?command=enablepoll |Enables the specified slave to poll for changes on the master.
+|http://_slave_host:port_/solr/_core_name_/replication?command=disablepoll |Disables the specified slave from polling for changes on the master.
+|http://_slave_host:port_/solr/_core_name_/replication?command=details |Retrieves configuration details and current status.
+|http://_host:port_/solr/_core_name_/replication?command=filelist&generation=<_generation-number_> |Retrieves a list of Lucene files present in the specified host's index. You can discover the generation number of the index by running the `indexversion` command.
+|http://_master_host:port_/solr/_core_name_/replication?command=backup a|
Creates a backup on master if there are committed index data in the server; otherwise, does nothing. This command is useful for making periodic backups.
supported request parameters:
@@ -258,10 +263,10 @@ supported request parameters:
* `name` : (optional) Backup name . The snapshot will be created in a directory called snapshot.<name> within the data directory of the core . By default the name is generated using date in `yyyyMMddHHmmssSSS` format. If `location` parameter is passed , that would be used instead of the data directory
* `location`: Backup location
-|http:// _master_host:port_ /solr/__core_name__/replication?command=deletebackup a|
+|http://_master_host:port_ /solr/_core_name_/replication?command=deletebackup a|
Delete any backup created using the `backup` command .
-request parameters:
+Request parameters:
* name: The name of the snapshot . A snapshot with the name snapshot.<name> must exist .If not, an error is thrown
* location: Location where the snapshot is created
@@ -277,7 +282,7 @@ The time required to optimize a master index can vary dramatically. A small inde
Distributing a newly optimized index may take only a few minutes or up to an hour or more, again depending on the size of the index and the performance capabilities of network connections and disks. During optimization the machine is under load and does not process queries very well. Given a schedule of updates being driven a few times an hour to the slaves, we cannot run an optimize with every committed snapshot.
-Copying an optimized index means that the *entire* index will need to be transferred during the next snappull. This is a large expense, but not nearly as huge as running the optimize everywhere. Consider this example: on a three-slave one-master configuration, distributing a newly-optimized index takes approximately 80 seconds __total__. Rolling the change across a tier would require approximately ten minutes per machine (or machine group). If this optimize were rolled across the query tier, and if each slave node being optimized were disabled and not receiving queries, a rollout would take at least twenty minutes and potentially as long as an hour and a half. Additionally, the files would need to be synchronized so that the _following_ the optimize, snappull would not think that the independently optimized files were different in any way. This would also leave the door open to independent corruption of indexes instead of each being a perfect copy of the master.
+Copying an optimized index means that the *entire* index will need to be transferred during the next snappull. This is a large expense, but not nearly as huge as running the optimize everywhere. Consider this example: on a three-slave one-master configuration, distributing a newly-optimized index takes approximately 80 seconds _total_. Rolling the change across a tier would require approximately ten minutes per machine (or machine group). If this optimize were rolled across the query tier, and if each slave node being optimized were disabled and not receiving queries, a rollout would take at least twenty minutes and potentially as long as an hour and a half. Additionally, the files would need to be synchronized so that the _following_ the optimize, snappull would not think that the independently optimized files were different in any way. This would also leave the door open to independent corruption of indexes instead of each being a perfect copy of the master.
Optimizing on the master allows for a straight-forward optimization operation. No query slaves need to be taken out of service. The optimized index can be distributed in the background as queries are being normally serviced. The optimization can occur at any time convenient to the application providing index updates.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/d77278df/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc b/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc
index e15746e..d8c84bb 100644
--- a/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc
+++ b/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc
@@ -2,7 +2,9 @@
:page-shortname: indexconfig-in-solrconfig
:page-permalink: indexconfig-in-solrconfig.html
-The `<indexConfig>` section of `solrconfig.xml` defines low-level behavior of the Lucene index writers. By default, the settings are commented out in the sample `solrconfig.xml` included with Solr, which means the defaults are used. In most cases, the defaults are fine.
+The `<indexConfig>` section of `solrconfig.xml` defines low-level behavior of the Lucene index writers.
+
+By default, the settings are commented out in the sample `solrconfig.xml` included with Solr, which means the defaults are used. In most cases, the defaults are fine.
[source,xml]
----
@@ -50,7 +52,11 @@ Controls whether newly written (and not yet merged) index segments should use th
[[IndexConfiginSolrConfig-mergePolicyFactory]]
=== `mergePolicyFactory`
-Defines how merging segments is done. The default in Solr is to use a `TieredMergePolicy`, which merges segments of approximately equal size, subject to an allowed number of segments per tier. Other policies available are the `LogByteSizeMergePolicy` and `LogDocMergePolicy`. For more information on these policies, please see {lucene-javadocs}/core/org/apache/lucene/index/MergePolicy.html[the MergePolicy javadocs].
+Defines how merging segments is done.
+
+The default in Solr is to use a `TieredMergePolicy`, which merges segments of approximately equal size, subject to an allowed number of segments per tier.
+
+Other policies available are the `LogByteSizeMergePolicy` and `LogDocMergePolicy`. For more information on these policies, please see {lucene-javadocs}/core/org/apache/lucene/index/MergePolicy.html[the MergePolicy javadocs].
[source,xml]
----
@@ -60,14 +66,15 @@ Defines how merging segments is done. The default in Solr is to use a `TieredMer
</mergePolicyFactory>
----
-// OLD_CONFLUENCE_ID: IndexConfiginSolrConfig-ControllingSegmentSizes:MergeFactors
-
-[[IndexConfiginSolrConfig-ControllingSegmentSizes_MergeFactors]]
=== Controlling Segment Sizes: Merge Factors
-The most common adjustment some folks make to the configuration of TieredMergePolicy (or LogByteSizeMergePolicy) are the "merge factors" to change how many segments should be merged at one time. For TieredMergePolicy, this is controlled by setting the `<int name="maxMergeAtOnce">` and `<int name="segmentsPerTier">` options, while LogByteSizeMergePolicy has a single `<int name="mergeFactor">` option (all of which default to "```10```").
+The most common adjustment users make to the configuration of TieredMergePolicy (or LogByteSizeMergePolicy) are the "merge factors" to change how many segments should be merged at one time.
+
+For TieredMergePolicy, this is controlled by setting the `<int name="maxMergeAtOnce">` and `<int name="segmentsPerTier">` options, while LogByteSizeMergePolicy has a single `<int name="mergeFactor">` option (all of which default to `10`).
-To understand why these options are important, consider what happens when an update is made to an index using LogByteSizeMergePolicy: Documents are always added to the most recently opened segment. When a segment fills up, a new segment is created and subsequent updates are placed there. If creating a new segment would cause the number of lowest-level segments to exceed the `mergeFactor` value, then all those segments are merged together to form a single large segment. Thus, if the merge factor is 10, each merge results in the creation of a single segment that is roughly ten times larger than each of its ten constituents. When there are 10 of these larger segments, then they in turn are merged into an even larger single segment. This process can continue indefinitely.
+To understand why these options are important, consider what happens when an update is made to an index using LogByteSizeMergePolicy: Documents are always added to the most recently opened segment. When a segment fills up, a new segment is created and subsequent updates are placed there.
+
+If creating a new segment would cause the number of lowest-level segments to exceed the `mergeFactor` value, then all those segments are merged together to form a single large segment. Thus, if the merge factor is 10, each merge results in the creation of a single segment that is roughly ten times larger than each of its ten constituents. When there are 10 of these larger segments, then they in turn are merged into an even larger single segment. This process can continue indefinitely.
When using TieredMergePolicy, the process is the same, but instead of a single `mergeFactor` value, the `segmentsPerTier` setting is used as the threshold to decide if a merge should happen, and the `maxMergeAtOnce` setting determines how many segments should be included in the merge.
@@ -116,12 +123,15 @@ When using Solr in for <<near-real-time-searching.adoc#near-real-time-searching,
[[IndexConfiginSolrConfig-CompoundFileSegments]]
== Compound File Segments
-Each Lucene segment is typically comprised of a dozen or so files. Lucene can be configured to bundle all of the files for a segment into a single compound file using a file extension of `.cfs`; it's an abbreviation for Compound File Segment. CFS segments may incur a minor performance hit for various reasons, depending on the runtime environment. For example, filesystem buffers are typically associated with open file descriptors, which may limit the total cache space available to each index. On systems where the number of open files allowed per process is limited, CFS may avoid hitting that limit. The open files limit might also be tunable for your OS with the Linux/Unix `ulimit` command, or something similar for other operating systems.
+Each Lucene segment is typically comprised of a dozen or so files. Lucene can be configured to bundle all of the files for a segment into a single compound file using a file extension of `.cfs`; it's an abbreviation for Compound File Segment.
+
+CFS segments may incur a minor performance hit for various reasons, depending on the runtime environment. For example, filesystem buffers are typically associated with open file descriptors, which may limit the total cache space available to each index.
+
+On systems where the number of open files allowed per process is limited, CFS may avoid hitting that limit. The open files limit might also be tunable for your OS with the Linux/Unix `ulimit` command, or something similar for other operating systems.
.CFS: New Segments vs Merged Segments
[NOTE]
====
-
To configure whether _newly written segments_ should use CFS, see the <<IndexConfiginSolrConfig-useCompoundFile,`useCompoundFile`>> setting described above. To configure whether _merged segments_ use CFS, review the Javadocs for your <<IndexConfiginSolrConfig-mergePolicyFactory,`mergePolicyFactory`>> .
Many <<IndexConfiginSolrConfig-MergingIndexSegments,Merge Policy>> implementations support `noCFSRatio` and `maxCFSSegmentSizeMB` settings with default values that prevent compound files from being used for large segments, but do use compound files for small segments.
@@ -165,7 +175,7 @@ The maximum time to wait for a write lock on an IndexWriter. The default is 1000
There are a few other parameters that may be important to configure for your implementation. These settings affect how or when updates are made to an index.
-[width="100%",cols="50%,50%",options="header",]
+[width="100%",options="header",]
|===
|Setting |Description
|reopenReaders |Controls if IndexReaders will be re-opened, instead of closed and then opened, which is often less efficient. The default is true.
@@ -177,8 +187,8 @@ There are a few other parameters that may be important to configure for your imp
----
<reopenReaders>true</reopenReaders>
<deletionPolicy class="solr.SolrDeletionPolicy">
- <str name="maxCommitsToKeep">1</str>
- <str name="maxOptimizedCommitsToKeep">0</str>
+ <str name="maxCommitsToKeep">1</str>
+ <str name="maxOptimizedCommitsToKeep">0</str>
<str name="maxCommitAge">1DAY</str>
</deletionPolicy>
<infoStream>false</infoStream>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/d77278df/solr/solr-ref-guide/src/indexing-and-basic-data-operations.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/indexing-and-basic-data-operations.adoc b/solr/solr-ref-guide/src/indexing-and-basic-data-operations.adoc
index dde6bfe..6d6d184 100644
--- a/solr/solr-ref-guide/src/indexing-and-basic-data-operations.adoc
+++ b/solr/solr-ref-guide/src/indexing-and-basic-data-operations.adoc
@@ -5,27 +5,27 @@
This section describes how Solr adds data to its index. It covers the following topics:
-* **<<introduction-to-solr-indexing.adoc#introduction-to-solr-indexing,Introduction to Solr Indexing>>**: An overview of Solr's indexing process.
+* *<<introduction-to-solr-indexing.adoc#introduction-to-solr-indexing,Introduction to Solr Indexing>>*: An overview of Solr's indexing process.
-* **<<post-tool.adoc#post-tool,Post Tool>>**: Information about using `post.jar` to quickly upload some content to your system.
+* *<<post-tool.adoc#post-tool,Post Tool>>*: Information about using `post.jar` to quickly upload some content to your system.
-* **<<uploading-data-with-index-handlers.adoc#uploading-data-with-index-handlers,Uploading Data with Index Handlers>>**: Information about using Solr's Index Handlers to upload XML/XSLT, JSON and CSV data.
+* *<<uploading-data-with-index-handlers.adoc#uploading-data-with-index-handlers,Uploading Data with Index Handlers>>*: Information about using Solr's Index Handlers to upload XML/XSLT, JSON and CSV data.
* *<<transforming-and-indexing-custom-json.adoc#transforming-and-indexing-custom-json,Transforming and Indexing Custom JSON>>* : Index any JSON of your choice
-* **<<uploading-data-with-solr-cell-using-apache-tika.adoc#uploading-data-with-solr-cell-using-apache-tika,Uploading Data with Solr Cell using Apache Tika>>**: Information about using the Solr Cell framework to upload data for indexing.
+* *<<uploading-data-with-solr-cell-using-apache-tika.adoc#uploading-data-with-solr-cell-using-apache-tika,Uploading Data with Solr Cell using Apache Tika>>*: Information about using the Solr Cell framework to upload data for indexing.
-* **<<uploading-structured-data-store-data-with-the-data-import-handler.adoc#uploading-structured-data-store-data-with-the-data-import-handler,Uploading Structured Data Store Data with the Data Import Handler>>**: Information about uploading and indexing data from a structured data store.
+* *<<uploading-structured-data-store-data-with-the-data-import-handler.adoc#uploading-structured-data-store-data-with-the-data-import-handler,Uploading Structured Data Store Data with the Data Import Handler>>*: Information about uploading and indexing data from a structured data store.
-* **<<updating-parts-of-documents.adoc#updating-parts-of-documents,Updating Parts of Documents>>**: Information about how to use atomic updates and optimistic concurrency with Solr.
+* *<<updating-parts-of-documents.adoc#updating-parts-of-documents,Updating Parts of Documents>>*: Information about how to use atomic updates and optimistic concurrency with Solr.
-* **<<detecting-languages-during-indexing.adoc#detecting-languages-during-indexing,Detecting Languages During Indexing>>**: Information about using language identification during the indexing process.
+* *<<detecting-languages-during-indexing.adoc#detecting-languages-during-indexing,Detecting Languages During Indexing>>*: Information about using language identification during the indexing process.
-* **<<de-duplication.adoc#de-duplication,De-Duplication>>**: Information about configuring Solr to mark duplicate documents as they are indexed.
+* *<<de-duplication.adoc#de-duplication,De-Duplication>>*: Information about configuring Solr to mark duplicate documents as they are indexed.
-* **<<content-streams.adoc#content-streams,Content Streams>>**: Information about streaming content to Solr Request Handlers.
+* *<<content-streams.adoc#content-streams,Content Streams>>*: Information about streaming content to Solr Request Handlers.
-* **<<uima-integration.adoc#uima-integration,UIMA Integration>>**: Information about integrating Solr with Apache's Unstructured Information Management Architecture (UIMA). UIMA lets you define custom pipelines of Analysis Engines that incrementally add metadata to your documents as annotations.
+* *<<uima-integration.adoc#uima-integration,UIMA Integration>>*: Information about integrating Solr with Apache's Unstructured Information Management Architecture (UIMA). UIMA lets you define custom pipelines of Analysis Engines that incrementally add metadata to your documents as annotations.
[[IndexingandBasicDataOperations-IndexingUsingClientAPIs]]
=== Indexing Using Client APIs
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/d77278df/solr/solr-ref-guide/src/indexupgrader-tool.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/indexupgrader-tool.adoc b/solr/solr-ref-guide/src/indexupgrader-tool.adoc
index 3a36cfa..1c3db45 100644
--- a/solr/solr-ref-guide/src/indexupgrader-tool.adoc
+++ b/solr/solr-ref-guide/src/indexupgrader-tool.adoc
@@ -8,7 +8,7 @@ The tool can be used from command line, or it can be instantiated and executed i
In a Solr distribution, the Lucene files are located in `./server/solr-webapp/webapp/WEB-INF/lib`. You will need to include the `lucene-core-<version>.jar` and `lucene-backwards-codecs-<version>.jar` on the classpath when running the tool.
-[source,text]
+[source,bash]
----
java -cp lucene-core-6.0.0.jar:lucene-backward-codecs-6.0.0.jar org.apache.lucene.index.IndexUpgrader [-delete-prior-commits] [-verbose] /path/to/index
----
@@ -19,7 +19,5 @@ Upgrading large indexes may take a long time. As a rule of thumb, the upgrade pr
[WARNING]
====
-
This tool may reorder documents if the index was partially upgraded before execution (e.g., documents were added). If your application relies on monotonicity of document IDs (which means that the order in which the documents were added to the index is preserved), do a full forceMerge instead.
-
====
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/d77278df/solr/solr-ref-guide/src/initparams-in-solrconfig.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/initparams-in-solrconfig.adoc b/solr/solr-ref-guide/src/initparams-in-solrconfig.adoc
index e9c15f7..f63c39d 100644
--- a/solr/solr-ref-guide/src/initparams-in-solrconfig.adoc
+++ b/solr/solr-ref-guide/src/initparams-in-solrconfig.adoc
@@ -4,9 +4,9 @@
The `<initParams>` section of `solrconfig.xml` allows you to define request handler parameters outside of the handler configuration.
-The use cases are
+There are a couple of use cases where this might be desired:
-* Some handlers are implicitly defined in code - see <<implicit-requesthandlers.adoc#implicit-requesthandlers,Implicit RequestHandlers>> - and there should be a way to add/append/override some of the implicitly defined properties
+* Some handlers are implicitly defined in code - see <<implicit-requesthandlers.adoc#implicit-requesthandlers,Implicit RequestHandlers>> - and there should be a way to add/append/override some of the implicitly defined properties.
* There are a few properties that are used across handlers. This helps you keep only a single definition of those properties and apply them over multiple handlers.
For example, if you want several of your search handlers to return the same list of fields, you can create an `<initParams>` section without having to define the same set of parameters in each request handler definition. If you have a single request handler that should return different fields, you can define the overriding parameters in individual `<requestHandler>` sections as usual.
@@ -30,9 +30,9 @@ The syntax and semantics are similar to that of a `<requestHandler>` . The follo
// TODO: This table has cells that won't work with PDF: https://github.com/ctargett/refguide-asciidoc-poc/issues/13
-[width="100%",cols="50%,50%",options="header",]
+[width="100%",options="header",]
|===
-|property |Description
+|Property |Description
|path |A comma-separated list of paths which will use the parameters. Wildcards can be used in paths to define nested paths, as described below.
|name a|
The name of this set of parameters. The name can be used directly in a requestHandler definition if a path is not explicitly named. If you give your `<initParams>` a name, you can refer to the params in a `<requestHandler>` that is not defined as a path.
@@ -49,7 +49,7 @@ For example, if an `<initParams>` section has the name "myParams", you can call
[[InitParamsinSolrConfig-Wildcards]]
=== Wildcards
-An `<initParams>` section can support wildcards to define nested paths that should use the parameters defined. A single asterisk (*) denotes that a nested path one level deeper should use the parameters. Double asterisks (**) denote all nested paths no matter how deep should use the parameters.
+An `<initParams>` section can support wildcards to define nested paths that should use the parameters defined. A single asterisk (\*) denotes that a nested path one level deeper should use the parameters. Double asterisks (**) denote all nested paths no matter how deep should use the parameters.
For example, if we have an `<initParams>` that looks like this:
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/d77278df/solr/solr-ref-guide/src/installing-solr.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/installing-solr.adoc b/solr/solr-ref-guide/src/installing-solr.adoc
index 483a45c..44a7993 100644
--- a/solr/solr-ref-guide/src/installing-solr.adoc
+++ b/solr/solr-ref-guide/src/installing-solr.adoc
@@ -2,11 +2,10 @@
:page-shortname: installing-solr
:page-permalink: installing-solr.html
-This section describes how to install Solr. You can install Solr in any system where a suitable Java Runtime Environment (JRE) is available, as detailed below. Currently this includes Linux, OS X, and Microsoft Windows. The instructions in this section should work for any platform, with a few exceptions for Windows as noted.
+This section describes how to install Solr.
-// OLD_CONFLUENCE_ID: InstallingSolr-GotJava?
+You can install Solr in any system where a suitable Java Runtime Environment (JRE) is available, as detailed below. Currently this includes Linux, OS X, and Microsoft Windows. The instructions in this section should work for any platform, with a few exceptions for Windows as noted.
-[[InstallingSolr-GotJava_]]
== Got Java?
You will need the Java Runtime Environment (JRE) version 1.8 or higher. At a command line, check your Java version like this:
@@ -21,12 +20,15 @@ Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
The exact output will vary, but you need to make sure you meet the minimum version requirement. We also recommend choosing a version that is not end-of-life from its vendor. If you don't have the required version, or if the java command is not found, download and install the latest version from Oracle at http://www.oracle.com/technetwork/java/javase/downloads/index.html.
-[[InstallingSolr-InstallingSolr]]
== Installing Solr
Solr is available from the Solr website at http://lucene.apache.org/solr/.
-For Linux/Unix/OSX systems, download the `.tgz` file. For Microsoft Windows systems, download the `.zip` file. When getting started, all you need to do is extract the Solr distribution archive to a directory of your choosing. When you're ready to setup Solr for a production environment, please refer to the instructions provided on the <<taking-solr-to-production.adoc#taking-solr-to-production,Taking Solr to Production>> page. To keep things simple for now, extract the Solr distribution archive to your local home directory, for instance on Linux, do:
+For Linux/Unix/OSX systems, download the `.tgz` file. For Microsoft Windows systems, download the `.zip` file.
+
+When getting started, all you need to do is extract the Solr distribution archive to a directory of your choosing. When you're ready to setup Solr for a production environment, please refer to the instructions provided on the <<taking-solr-to-production.adoc#taking-solr-to-production,Taking Solr to Production>> page.
+
+To keep things simple for now, extract the Solr distribution archive to your local home directory, for instance on Linux, do:
[source,plain]
----
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/d77278df/solr/solr-ref-guide/src/introduction-to-scaling-and-distribution.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/introduction-to-scaling-and-distribution.adoc b/solr/solr-ref-guide/src/introduction-to-scaling-and-distribution.adoc
index c399db6..9eb4313 100644
--- a/solr/solr-ref-guide/src/introduction-to-scaling-and-distribution.adoc
+++ b/solr/solr-ref-guide/src/introduction-to-scaling-and-distribution.adoc
@@ -2,7 +2,9 @@
:page-shortname: introduction-to-scaling-and-distribution
:page-permalink: introduction-to-scaling-and-distribution.html
-Both Lucene and Solr were designed to scale to support large implementations with minimal custom coding. This section covers:
+Both Lucene and Solr were designed to scale to support large implementations with minimal custom coding.
+
+This section covers:
* <<distributed-search-with-index-sharding.adoc#distributed-search-with-index-sharding,distributing>> an index across multiple servers
* <<index-replication.adoc#index-replication,replicating>> an index on multiple servers
@@ -10,18 +12,14 @@ Both Lucene and Solr were designed to scale to support large implementations wit
If you need full scale distribution of indexes and queries, as well as replication, load balancing and failover, you may want to use SolrCloud. Full details on configuring and using SolrCloud is available in the section <<solrcloud.adoc#solrcloud,SolrCloud>>.
-// OLD_CONFLUENCE_ID: IntroductiontoScalingandDistribution-WhatProblemDoesDistributionSolve?
-
-[[IntroductiontoScalingandDistribution-WhatProblemDoesDistributionSolve_]]
== What Problem Does Distribution Solve?
If searches are taking too long or the index is approaching the physical limitations of its machine, you should consider distributing the index across two or more Solr servers.
-To distribute an index, you divide the index into partitions called shards, each of which runs on a separate machine. Solr then partitions searches into sub-searches, which run on the individual shards, reporting results collectively. The architectural details underlying index sharding are invisible to end users, who simply experience faster performance on queries against very large indexes.
+To distribute an index, you divide the index into partitions called shards, each of which runs on a separate machine. Solr then partitions searches into sub-searches, which run on the individual shards, reporting results collectively.
-// OLD_CONFLUENCE_ID: IntroductiontoScalingandDistribution-WhatProblemDoesReplicationSolve?
+The architectural details underlying index sharding are invisible to end users, who simply experience faster performance on queries against very large indexes.
-[[IntroductiontoScalingandDistribution-WhatProblemDoesReplicationSolve_]]
== What Problem Does Replication Solve?
Replicating an index is useful when:
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/d77278df/solr/solr-ref-guide/src/introduction-to-solr-indexing.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/introduction-to-solr-indexing.adoc b/solr/solr-ref-guide/src/introduction-to-solr-indexing.adoc
index 11c26f5..80f4cfd 100644
--- a/solr/solr-ref-guide/src/introduction-to-solr-indexing.adoc
+++ b/solr/solr-ref-guide/src/introduction-to-solr-indexing.adoc
@@ -2,7 +2,9 @@
:page-shortname: introduction-to-solr-indexing
:page-permalink: introduction-to-solr-indexing.html
-This section describes the process of indexing: adding content to a Solr index and, if necessary, modifying that content or deleting it. By adding content to an index, we make it searchable by Solr.
+This section describes the process of indexing: adding content to a Solr index and, if necessary, modifying that content or deleting it.
+
+By adding content to an index, we make it searchable by Solr.
A Solr index can accept data from many different sources, including XML files, comma-separated value (CSV) files, data extracted from tables in a database, and files in common file formats such as Microsoft Word or PDF.
@@ -12,7 +14,7 @@ Here are the three most common ways of loading data into a Solr index:
* Uploading XML files by sending HTTP requests to the Solr server from any environment where such requests can be generated.
-* Writing a custom Java application to ingest data through Solr's Java Client API (which is described in more detail in <<client-apis.adoc#client-apis,Client APIs>>. Using the Java API may be the best choice if you're working with an application, such as a Content Management System (CMS), that offers a Java API.
+* Writing a custom Java application to ingest data through Solr's Java Client API (which is described in more detail in <<client-apis.adoc#client-apis,Client APIs>>). Using the Java API may be the best choice if you're working with an application, such as a Content Management System (CMS), that offers a Java API.
Regardless of the method used to ingest data, there is a common basic data structure for data being fed into a Solr index: a _document_ containing multiple _fields,_ each with a _name_ and containing _content,_ which may be empty. One of the fields is usually designated as a unique ID field (analogous to a primary key in a database), although the use of a unique ID field is not strictly required by Solr.
@@ -32,9 +34,7 @@ Many of the instructions and examples in this section make use of the `curl` uti
[IMPORTANT]
====
-
Using `curl` or other command line tools for posting data is just fine for examples or tests, but it's not the recommended method for achieving the best performance for updates in production environments. You will achieve better performance with Solr Cell or the other methods described in this section.
Instead of `curl`, you can use utilities such as GNU `wget` (http://www.gnu.org/software/wget/) or manage GETs and POSTS with Perl, although the command line options will differ.
-
====