You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by is...@apache.org on 2017/07/29 21:59:47 UTC
[10/28] lucene-solr:jira/solr-6630: Merging master
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8d00e53b/solr/solr-ref-guide/src/graph-traversal.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/graph-traversal.adoc b/solr/solr-ref-guide/src/graph-traversal.adoc
index 007019b..a23b32e 100644
--- a/solr/solr-ref-guide/src/graph-traversal.adoc
+++ b/solr/solr-ref-guide/src/graph-traversal.adoc
@@ -31,7 +31,6 @@ The `nodes` function can be combined with the `scoreNodes` function to provide r
This document assumes a basic understanding of graph terminology and streaming expressions. You can begin exploring graph traversal concepts with this https://en.wikipedia.org/wiki/Graph_traversal[Wikipedia article]. More details about streaming expressions are available in this Guide, in the section <<streaming-expressions.adoc#streaming-expressions,Streaming Expressions>>.
====
-[[GraphTraversal-BasicSyntax]]
== Basic Syntax
We'll start with the most basic syntax and slowly build up more complexity. The most basic syntax for `nodes` is:
@@ -161,7 +160,6 @@ When scattering both branches and leaves the output would like this:
Now the level 0 root node is included in the output.
-[[GraphTraversal-Aggregations]]
== Aggregations
`nodes` also supports aggregations. For example:
@@ -182,8 +180,7 @@ Edges are uniqued as part of the traversal so the count will *not* reflect the n
The aggregation functions supported are `count(*)`, `sum(field)`, `min(field)`, `max(field)`, and `avg(field)`. The fields being aggregated should be present in the edges collected during the traversal. Later examples (below) will show aggregations can be a powerful tool for providing recommendations and limiting the scope of traversals.
-[[GraphTraversal-Nestingnodesfunctions]]
-== Nesting nodes functions
+== Nesting nodes Functions
The `nodes` function can be nested to traverse deeper into the graph. For example:
@@ -207,14 +204,12 @@ Put more simply, the inner expression gathers all the people that "\johndoe@apac
This construct of nesting `nodes` functions is the basic technique for doing a controlled traversal through the graph.
-[[GraphTraversal-CycleDetection]]
== Cycle Detection
The `nodes` function performs cycle detection across the entire traversal. This ensures that nodes that have already been visited are not traversed again. Cycle detection is important for both limiting the size of traversals and gathering accurate aggregations. Without cycle detection the size of the traversal could grow exponentially with each hop in the traversal. With cycle detection only new nodes encountered are traversed.
Cycle detection *does not* cross collection boundaries. This is because internally the collection name is part of the node ID. For example the node ID "\johndoe@apache.org", is really `emails/johndoe@apache.org`. When traversing to another collection "\johndoe@apache.org" will be traversed.
-[[GraphTraversal-FilteringtheTraversal]]
== Filtering the Traversal
Each level in the traversal can be filtered with a filter query. For example:
@@ -229,7 +224,6 @@ nodes(emails,
In the example above only emails that match the filter query will be included in the traversal. Any Solr query can be included here. So you can do fun things like <<spatial-search.adoc#spatial-search,geospatial queries>>, apply any of the available <<query-syntax-and-parsing.adoc#query-syntax-and-parsing,query parsers>>, or even write custom query parsers to limit the traversal.
-[[GraphTraversal-RootStreams]]
== Root Streams
Any streaming expression can be used to provide the root nodes for a traversal. For example:
@@ -246,7 +240,6 @@ The example above provides the root nodes through a search expression. You can a
Notice that the `walk` parameter maps a field from the tuples generated by the inner stream. In this case it maps the `to` field from the inner stream to the `from` field.
-[[GraphTraversal-SkippingHighFrequencyNodes]]
== Skipping High Frequency Nodes
It's often desirable to skip traversing high frequency nodes in the graph. This is similar in nature to a search term stop list. The best way to describe this is through an example use case.
@@ -277,7 +270,6 @@ The `nodes` function has the `maxDocFreq` param to allow for filtering out high
In the example above, the inner search expression searches the `logs` collection and returning all the articles viewed by "user1". The outer `nodes` expression takes all the articles emitted from the inner search expression and finds all the records in the logs collection for those articles. It then gathers and aggregates the users that have read the articles. The `maxDocFreq` parameter limits the articles returned to those that appear in no more then 10,000 log records (per shard). This guards against returning articles that have been viewed by millions of users.
-[[GraphTraversal-TrackingtheTraversal]]
== Tracking the Traversal
By default the `nodes` function only tracks enough information to do cycle detection. This provides enough information to output the nodes and aggregations in the graph.
@@ -298,7 +290,6 @@ nodes(emails,
gather="to")
----
-[[GraphTraversal-Cross-CollectionTraversals]]
== Cross-Collection Traversals
Nested `nodes` functions can operate on different SolrCloud collections. This allow traversals to "walk" from one collection to another to gather nodes. Cycle detection does not cross collection boundaries, so nodes collected in one collection will be traversed in a different collection. This was done deliberately to support cross-collection traversals. Note that the output from a cross-collection traversal will likely contain duplicate nodes with different collection attributes.
@@ -320,7 +311,6 @@ nodes(logs,
The example above finds all people who sent emails with a body that contains "solr rocks". It then finds all the people these people have emailed. Then it traverses to the logs collection and gathers all the content IDs that these people have edited.
-[[GraphTraversal-CombiningnodesWithOtherStreamingExpressions]]
== Combining nodes With Other Streaming Expressions
The `nodes` function can act as both a stream source and a stream decorator. The connection with the wider stream expression library provides tremendous power and flexibility when performing graph traversals. Here is an example of using the streaming expression library to intersect two friend networks:
@@ -348,10 +338,8 @@ The `nodes` function can act as both a stream source and a stream decorator. The
The example above gathers two separate friend networks, one rooted with "\johndoe@apache.org" and another rooted with "\janedoe@apache.org". The friend networks are then sorted by the `node` field, and intersected. The resulting node set will be the intersection of the two friend networks.
-[[GraphTraversal-SampleUseCases]]
-== Sample Use Cases
+== Sample Use Cases for Graph Traversal
-[[GraphTraversal-CalculateMarketBasketCo-occurrence]]
=== Calculate Market Basket Co-occurrence
It is often useful to know which products are most frequently purchased with a particular product. This example uses a simple market basket table (indexed in Solr) to store past shopping baskets. The schema for the table is very simple with each row containing a `basketID` and a `productID`. This can be seen as a graph with each row in the table representing an edge. And it can be traversed very quickly to calculate basket co-occurrence, even when the graph contains billions of edges.
@@ -378,15 +366,13 @@ Let's break down exactly what this traversal is doing.
In a nutshell this expression finds the products that most frequently co-occur with product "ABC" in past shopping baskets.
-[[GraphTraversal-UsingthescoreNodesFunctiontoMakeaRecommendation]]
=== Using the scoreNodes Function to Make a Recommendation
-This use case builds on the market basket example <<GraphTraversal-CalculateMarketBasketCo-occurrence,above>> that calculates which products co-occur most frequently with productID:ABC. The ranked co-occurrence counts provide candidates for a recommendation. The `scoreNodes` function can be used to score the candidates to find the best recommendation.
+This use case builds on the market basket example <<Calculate Market Basket Co-occurrence,above>> that calculates which products co-occur most frequently with productID:ABC. The ranked co-occurrence counts provide candidates for a recommendation. The `scoreNodes` function can be used to score the candidates to find the best recommendation.
Before diving into the syntax of the `scoreNodes` function it's useful to understand why the raw co-occurrence counts may not produce the best recommendation. The reason is that raw co-occurrence counts favor items that occur frequently across all baskets. A better recommendation would find the product that has the most significant relationship with productID ABC. The `scoreNodes` function uses a term frequency-inverse document frequency (TF-IDF) algorithm to find the most significant relationship.
-[[GraphTraversal-HowItWorks]]
-==== *How It Works*
+==== How scoreNodes Works
The `scoreNodes` function assigns a score to each node emitted by the nodes expression. By default the `scoreNodes` function uses the `count(*)` aggregation, which is the co-occurrence count, as the TF value. The IDF value for each node is fetched from the collection where the node was gathered. Each node is then scored using the TF*IDF formula, which provides a boost to nodes with a lower frequency across all market baskets.
@@ -394,8 +380,7 @@ Combining the co-occurrence count with the IDF provides a score that shows how i
The `scoreNodes` function adds the score to each node in the `nodeScore` field.
-[[GraphTraversal-ExampleSyntax]]
-==== *Example Syntax*
+==== Example scoreNodes Syntax
[source,plain]
----
@@ -417,7 +402,6 @@ This example builds on the earlier example "Calculate market basket co-occurrenc
. The `scoreNodes` function then assigns a score to the candidates based on the TF*IDF of each node.
. The outer `top` expression selects the highest scoring node. This is the recommendation.
-[[GraphTraversal-RecommendContentBasedonCollaborativeFilter]]
=== Recommend Content Based on Collaborative Filter
In this example we'll recommend content for a user based on a collaborative filter. This recommendation is made using log records that contain the `userID` and `articleID` and the action performed. In this scenario each log record can be viewed as an edge in a graph. The userID and articleID are the nodes and the action is an edge property used to filter the traversal.
@@ -458,7 +442,6 @@ Note that it skips high frequency nodes using the `maxDocFreq` param to filter o
Any article selected in step 1 (user1 reading list), will not appear in this step due to cycle detection. So this step returns the articles read by the users with the most similar readings habits to "user1" that "user1" has not read yet. It also counts the number of times each article has been read across this user group.
. The outer `top` expression takes the top articles emitted from step 4. This is the recommendation.
-[[GraphTraversal-ProteinPathwayTraversal]]
=== Protein Pathway Traversal
In recent years, scientists have become increasingly able to rationally design drugs that target the mutated proteins, called oncogenes, responsible for some cancers. Proteins typically act through long chains of chemical interactions between multiple proteins, called pathways, and, while the oncogene in the pathway may not have a corresponding drug, another protein in the pathway may. Graph traversal on a protein collection that records protein interactions and drugs may yield possible candidates. (Thanks to Lewis Geer of the NCBI, for providing this example).
@@ -481,7 +464,6 @@ Let's break down exactly what this traversal is doing.
. The outer `nodes` expression also works with the `proteins` collection. It gathers all the drugs that correspond to proteins emitted from step 1.
. Using this stepwise approach you can gather the drugs along the pathway of interactions any number of steps away from the root protein.
-[[GraphTraversal-ExportingGraphMLtoSupportGraphVisualization]]
== Exporting GraphML to Support Graph Visualization
In the examples above, the `nodes` expression was sent to Solr's `/stream` handler like any other streaming expression. This approach outputs the nodes in the same JSON tuple format as other streaming expressions so that it can be treated like any other streaming expression. You can use the `/stream` handler when you need to operate directly on the tuples, such as in the recommendation use cases above.
@@ -496,8 +478,7 @@ There are a few things to keep mind when exporting a graph in GraphML:
. The `/graph` handler currently accepts an arbitrarily complex streaming expression which includes a `nodes` expression. If the streaming expression doesn't include a `nodes` expression, the `/graph` handler will not properly output GraphML.
. The `/graph` handler currently accepts a single arbitrarily complex, nested `nodes` expression per request. This means you cannot send in a streaming expression that joins or intersects the node sets from multiple `nodes` expressions. The `/graph` handler does support any level of nesting within a single `nodes` expression. The `/stream` handler does support joining and intersecting node sets, but the `/graph` handler currently does not.
-[[GraphTraversal-SampleRequest]]
-=== Sample Request
+=== Sample GraphML Request
[source,bash]
----
@@ -512,7 +493,6 @@ curl --data-urlencode 'expr=nodes(enron_emails,
gather="to")' http://localhost:8983/solr/enron_emails/graph
----
-[[GraphTraversal-SampleGraphMLOutput]]
=== Sample GraphML Output
[source,xml]
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8d00e53b/solr/solr-ref-guide/src/hadoop-authentication-plugin.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/hadoop-authentication-plugin.adoc b/solr/solr-ref-guide/src/hadoop-authentication-plugin.adoc
index 1c17fbc..7ed0a15 100644
--- a/solr/solr-ref-guide/src/hadoop-authentication-plugin.adoc
+++ b/solr/solr-ref-guide/src/hadoop-authentication-plugin.adoc
@@ -30,7 +30,7 @@ For some of the authentication schemes (e.g., Kerberos), Solr provides a native
There are two plugin classes:
-* `HadoopAuthPlugin`: This can be used with standalone Solr as well as Solrcloud with <<authentication-and-authorization-plugins.adoc#AuthenticationandAuthorizationPlugins-PKI,PKI authentication>> for internode communication.
+* `HadoopAuthPlugin`: This can be used with standalone Solr as well as Solrcloud with <<authentication-and-authorization-plugins.adoc#securing-inter-node-requests,PKI authentication>> for internode communication.
* `ConfigurableInternodeAuthHadoopPlugin`: This is an extension of HadoopAuthPlugin that allows you to configure the authentication scheme for internode communication.
[TIP]
@@ -38,7 +38,6 @@ There are two plugin classes:
For most SolrCloud or standalone Solr setups, the `HadoopAuthPlugin` should suffice.
====
-[[HadoopAuthenticationPlugin-PluginConfiguration]]
== Plugin Configuration
`class`::
@@ -70,11 +69,8 @@ Configures proxy users for the underlying Hadoop authentication mechanism. This
`clientBuilderFactory`:: No |
The `HttpClientBuilderFactory` implementation used for the Solr internal communication. Only applicable for `ConfigurableInternodeAuthHadoopPlugin`.
-
-[[HadoopAuthenticationPlugin-ExampleConfigurations]]
== Example Configurations
-[[HadoopAuthenticationPlugin-KerberosAuthenticationusingHadoopAuthenticationPlugin]]
=== Kerberos Authentication using Hadoop Authentication Plugin
This example lets you configure Solr to use Kerberos Authentication, similar to how you would use the <<kerberos-authentication-plugin.adoc#kerberos-authentication-plugin,Kerberos Authentication Plugin>>.
@@ -105,7 +101,6 @@ To setup this plugin, use the following in your `security.json` file.
}
----
-[[HadoopAuthenticationPlugin-SimpleAuthenticationwithDelegationTokens]]
=== Simple Authentication with Delegation Tokens
Similar to the previous example, this is an example of setting up a Solr cluster that uses delegation tokens. Refer to the parameters in the Hadoop authentication library's https://hadoop.apache.org/docs/stable/hadoop-auth/Configuration.html[documentation] or refer to the section <<kerberos-authentication-plugin.adoc#kerberos-authentication-plugin,Kerberos Authentication Plugin>> for further details. Please note that this example does not use Kerberos and the requests made to Solr must contain valid delegation tokens.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8d00e53b/solr/solr-ref-guide/src/highlighting.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/highlighting.adoc b/solr/solr-ref-guide/src/highlighting.adoc
index b0d094d..137eef4 100644
--- a/solr/solr-ref-guide/src/highlighting.adoc
+++ b/solr/solr-ref-guide/src/highlighting.adoc
@@ -24,7 +24,6 @@ The fragments are included in a special section of the query response (the `high
Highlighting is extremely configurable, perhaps more than any other part of Solr. There are many parameters each for fragment sizing, formatting, ordering, backup/alternate behavior, and more options that are hard to categorize. Nonetheless, highlighting is very simple to use.
-[[Highlighting-Usage]]
== Usage
=== Common Highlighter Parameters
@@ -36,7 +35,7 @@ Use this parameter to enable or disable highlighting. The default is `false`. If
`hl.method`::
The highlighting implementation to use. Acceptable values are: `unified`, `original`, `fastVector`. The default is `original`.
+
-See the <<Highlighting-ChoosingaHighlighter,Choosing a Highlighter>> section below for more details on the differences between the available highlighters.
+See the <<Choosing a Highlighter>> section below for more details on the differences between the available highlighters.
`hl.fl`::
Specifies a list of fields to highlight. Accepts a comma- or space-delimited list of fields for which Solr should generate highlighted snippets.
@@ -92,7 +91,6 @@ The default is `51200` characters.
There are more parameters supported as well depending on the highlighter (via `hl.method`) chosen.
-[[Highlighting-HighlightingintheQueryResponse]]
=== Highlighting in the Query Response
In the response to a query, Solr includes highlighting data in a section separate from the documents. It is up to a client to determine how to process this response and display the highlights to users.
@@ -102,7 +100,7 @@ Using the example documents included with Solr, we can see how this might work:
In response to a query such as:
[source,text]
-http://localhost:8983/solr/gettingstarted/select?hl=on&q=apple&wt=json&hl.fl=manu&fl=id,name,manu,cat
+http://localhost:8983/solr/gettingstarted/select?hl=on&q=apple&hl.fl=manu&fl=id,name,manu,cat
we get a response such as this (truncated slightly for space):
@@ -136,7 +134,6 @@ Note the two sections `docs` and `highlighting`. The `docs` section contains the
The `highlighting` section includes the ID of each document, and the field that contains the highlighted portion. In this example, we used the `hl.fl` parameter to say we wanted query terms highlighted in the "manu" field. When there is a match to the query term in that field, it will be included for each document ID in the list.
-[[Highlighting-ChoosingaHighlighter]]
== Choosing a Highlighter
Solr provides a `HighlightComponent` (a `SearchComponent`) and it's in the default list of components for search handlers. It offers a somewhat unified API over multiple actual highlighting implementations (or simply "highlighters") that do the business of highlighting.
@@ -173,7 +170,6 @@ The Unified Highlighter is exclusively configured via search parameters. In cont
In addition to further information below, more information can be found in the {solr-javadocs}/solr-core/org/apache/solr/highlight/package-summary.html[Solr javadocs].
-[[Highlighting-SchemaOptionsandPerformanceConsiderations]]
=== Schema Options and Performance Considerations
Fundamental to the internals of highlighting are detecting the _offsets_ of the individual words that match the query. Some of the highlighters can run the stored text through the analysis chain defined in the schema, some can look them up from _postings_, and some can look them up from _term vectors._ These choices have different trade-offs:
@@ -198,7 +194,6 @@ This is definitely the fastest option for highlighting wildcard queries on large
+
This adds substantial weight to the index – similar in size to the compressed stored text. If you are using the Unified Highlighter then this is not a recommended configuration since it's slower and heavier than postings with light term vectors. However, this could make sense if full term vectors are already needed for another use-case.
-[[Highlighting-TheUnifiedHighlighter]]
== The Unified Highlighter
The Unified Highlighter supports these following additional parameters to the ones listed earlier:
@@ -243,7 +238,6 @@ Indicates which character to break the text on. Use only if you have defined `hl
This is useful when the text has already been manipulated in advance to have a special delineation character at desired highlight passage boundaries. This character will still appear in the text as the last character of a passage.
-[[Highlighting-TheOriginalHighlighter]]
== The Original Highlighter
The Original Highlighter supports these following additional parameters to the ones listed earlier:
@@ -314,7 +308,6 @@ If this may happen and you know you don't need them for highlighting (i.e. your
The Original Highlighter has a plugin architecture that enables new functionality to be registered in `solrconfig.xml`. The "```techproducts```" configset shows most of these settings explicitly. You can use it as a guide to provide your own components to include a `SolrFormatter`, `SolrEncoder`, and `SolrFragmenter.`
-[[Highlighting-TheFastVectorHighlighter]]
== The FastVector Highlighter
The FastVector Highlighter (FVH) can be used in conjunction with the Original Highlighter if not all fields should be highlighted with the FVH. In such a mode, set `hl.method=original` and `f.yourTermVecField.hl.method=fastVector` for all fields that should use the FVH. One annoyance to keep in mind is that the Original Highlighter uses `hl.simple.pre` whereas the FVH (and other highlighters) use `hl.tag.pre`.
@@ -349,15 +342,12 @@ The maximum number of phrases to analyze when searching for the highest-scoring
`hl.multiValuedSeparatorChar`::
Text to use to separate one value from the next for a multi-valued field. The default is " " (a space).
-
-[[Highlighting-UsingBoundaryScannerswiththeFastVectorHighlighter]]
=== Using Boundary Scanners with the FastVector Highlighter
The FastVector Highlighter will occasionally truncate highlighted words. To prevent this, implement a boundary scanner in `solrconfig.xml`, then use the `hl.boundaryScanner` parameter to specify the boundary scanner for highlighting.
Solr supports two boundary scanners: `breakIterator` and `simple`.
-[[Highlighting-ThebreakIteratorBoundaryScanner]]
==== The breakIterator Boundary Scanner
The `breakIterator` boundary scanner offers excellent performance right out of the box by taking locale and boundary type into account. In most cases you will want to use the `breakIterator` boundary scanner. To implement the `breakIterator` boundary scanner, add this code to the `highlighting` section of your `solrconfig.xml` file, adjusting the type, language, and country values as appropriate to your application:
@@ -375,7 +365,6 @@ The `breakIterator` boundary scanner offers excellent performance right out of t
Possible values for the `hl.bs.type` parameter are WORD, LINE, SENTENCE, and CHARACTER.
-[[Highlighting-ThesimpleBoundaryScanner]]
==== The simple Boundary Scanner
The `simple` boundary scanner scans term boundaries for a specified maximum character value (`hl.bs.maxScan`) and for common delimiters such as punctuation marks (`hl.bs.chars`). The `simple` boundary scanner may be useful for some custom To implement the `simple` boundary scanner, add this code to the `highlighting` section of your `solrconfig.xml` file, adjusting the values as appropriate to your application:
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8d00e53b/solr/solr-ref-guide/src/how-solrcloud-works.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/how-solrcloud-works.adoc b/solr/solr-ref-guide/src/how-solrcloud-works.adoc
index 519a888..5e364ce 100644
--- a/solr/solr-ref-guide/src/how-solrcloud-works.adoc
+++ b/solr/solr-ref-guide/src/how-solrcloud-works.adoc
@@ -27,13 +27,11 @@ The following sections cover provide general information about how various SolrC
If you are already familiar with SolrCloud concepts and basic functionality, you can skip to the section covering <<solrcloud-configuration-and-parameters.adoc#solrcloud-configuration-and-parameters,SolrCloud Configuration and Parameters>>.
-[[HowSolrCloudWorks-KeySolrCloudConcepts]]
== Key SolrCloud Concepts
A SolrCloud cluster consists of some "logical" concepts layered on top of some "physical" concepts.
-[[HowSolrCloudWorks-Logical]]
-=== Logical
+=== Logical Concepts
* A Cluster can host multiple Collections of Solr Documents.
* A collection can be partitioned into multiple Shards, which contain a subset of the Documents in the Collection.
@@ -41,8 +39,7 @@ A SolrCloud cluster consists of some "logical" concepts layered on top of some "
** The theoretical limit to the number of Documents that Collection can reasonably contain.
** The amount of parallelization that is possible for an individual search request.
-[[HowSolrCloudWorks-Physical]]
-=== Physical
+=== Physical Concepts
* A Cluster is made up of one or more Solr Nodes, which are running instances of the Solr server process.
* Each Node can host multiple Cores.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8d00e53b/solr/solr-ref-guide/src/how-to-contribute.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/how-to-contribute.adoc b/solr/solr-ref-guide/src/how-to-contribute.adoc
new file mode 100644
index 0000000..a590b3a
--- /dev/null
+++ b/solr/solr-ref-guide/src/how-to-contribute.adoc
@@ -0,0 +1,51 @@
+= How to Contribute to Solr Documentation
+:page-tocclass: right
+:page-toclevels: 2
+:page-shortname: how-to-contribute
+:page-permalink: how-to-contribute.html
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+The Lucene/Solr project has made it easy for anyone to contribute to the Solr Reference Guide with a patch.
+
+The Guide is written in simple AsciiDoc-formatted files, and the source lives in the main Lucene/Solr source repository, right alongside the code.
+
+// This section only appears in the PDF version of the Ref Guide.
+ifdef::backend-pdf[]
+See the latest information on how to contribute to documentation online at https://lucene.apache.org/solr/guide/meta-docs.html.
+endif::[]
+
+// This section only appears in the HTML version of the Ref Guide.
+ifdef::backend-html5[]
+The following sections give an overview of how to work with AsciiDoc-format files, how our documentation is generated, and how to do a Ref Guide release.
+
+Once you have a patch you'd like to contribute, you can submit it with a JIRA issue, the same as any other patch. See also the Solr wiki section on https://wiki.apache.org/solr/HowToContribute[How To Contribute] for information on submitting patches.
+
+== Working with AsciiDoc Files
+include::meta-docs/asciidoc-syntax.adoc[leveloffset=+2]
+
+include::meta-docs/editing-tools.adoc[leveloffset=+2]
+
+== Modifying the Output Formats
+The Solr Reference Guide is published in two formats, HTML and PDF. Different tools are used for each.
+include::meta-docs/jekyll.adoc[leveloffset=+2]
+
+include::meta-docs/pdf.adoc[leveloffset=+2]
+
+== Building & Publishing the Guide
+include::meta-docs/publish.adoc[leveloffset=+2]
+endif::[]
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8d00e53b/solr/solr-ref-guide/src/implicit-requesthandlers.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/implicit-requesthandlers.adoc b/solr/solr-ref-guide/src/implicit-requesthandlers.adoc
index 3c87f8c..c10d93b 100644
--- a/solr/solr-ref-guide/src/implicit-requesthandlers.adoc
+++ b/solr/solr-ref-guide/src/implicit-requesthandlers.adoc
@@ -20,7 +20,6 @@
Solr ships with many out-of-the-box RequestHandlers, which are called implicit because they are not configured in `solrconfig.xml`.
-[[ImplicitRequestHandlers-ListofImplicitlyAvailableEndpoints]]
== List of Implicitly Available Endpoints
// TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
@@ -44,19 +43,18 @@ Solr ships with many out-of-the-box RequestHandlers, which are called implicit b
|`/debug/dump` |{solr-javadocs}/solr-core/org/apache/solr/handler/DumpRequestHandler.html[DumpRequestHandler] |`_DEBUG_DUMP` |Echo the request contents back to the client.
|<<exporting-result-sets.adoc#exporting-result-sets,`/export`>> |{solr-javadocs}/solr-core/org/apache/solr/handler/component/SearchHandler.html[SearchHandler] |`_EXPORT` |Export full sorted result sets.
|<<realtime-get.adoc#realtime-get,`/get`>> |{solr-javadocs}/solr-core/org/apache/solr/handler/RealTimeGetHandler.html[RealTimeGetHandler] |`_GET` |Real-time get: low-latency retrieval of the latest version of a document.
-|<<graph-traversal.adoc#GraphTraversal-ExportingGraphMLtoSupportGraphVisualization,`/graph`>> |{solr-javadocs}/solr-core/org/apache/solr/handler/GraphHandler.html[GraphHandler] |`_ADMIN_GRAPH` |Return http://graphml.graphdrawing.org/[GraphML] formatted output from a <<graph-traversal.adoc#graph-traversal,`gather` `Nodes` streaming expression>>.
+|<<graph-traversal.adoc#exporting-graphml-to-support-graph-visualization,`/graph`>> |{solr-javadocs}/solr-core/org/apache/solr/handler/GraphHandler.html[GraphHandler] |`_ADMIN_GRAPH` |Return http://graphml.graphdrawing.org/[GraphML] formatted output from a <<graph-traversal.adoc#graph-traversal,`gather` `Nodes` streaming expression>>.
|<<index-replication.adoc#index-replication,`/replication`>> |{solr-javadocs}/solr-core/org/apache/solr/handler/ReplicationHandler.html[ReplicationHandler] |`_REPLICATION` |Replicate indexes for SolrCloud recovery and Master/Slave index distribution.
|<<schema-api.adoc#schema-api,`/schema`>> |{solr-javadocs}/solr-core/org/apache/solr/handler/SchemaHandler.html[SchemaHandler] |`_SCHEMA` |Retrieve/modify Solr schema.
|<<parallel-sql-interface.adoc#sql-request-handler,`/sql`>> |{solr-javadocs}/solr-core/org/apache/solr/handler/SQLHandler.html[SQLHandler] |`_SQL` |Front end of the Parallel SQL interface.
-|<<streaming-expressions.adoc#StreamingExpressions-StreamingRequestsandResponses,`/stream`>> |{solr-javadocs}/solr-core/org/apache/solr/handler/StreamHandler.html[StreamHandler] |`_STREAM` |Distributed stream processing.
-|<<the-terms-component.adoc#TheTermsComponent-UsingtheTermsComponentinaRequestHandler,`/terms`>> |{solr-javadocs}/solr-core/org/apache/solr/handler/component/SearchHandler.html[SearchHandler] |`_TERMS` |Return a field's indexed terms and the number of documents containing each term.
+|<<streaming-expressions.adoc#streaming-requests-and-responses,`/stream`>> |{solr-javadocs}/solr-core/org/apache/solr/handler/StreamHandler.html[StreamHandler] |`_STREAM` |Distributed stream processing.
+|<<the-terms-component.adoc#using-the-terms-component-in-a-request-handler,`/terms`>> |{solr-javadocs}/solr-core/org/apache/solr/handler/component/SearchHandler.html[SearchHandler] |`_TERMS` |Return a field's indexed terms and the number of documents containing each term.
|<<uploading-data-with-index-handlers.adoc#uploading-data-with-index-handlers,`/update`>> |{solr-javadocs}/solr-core/org/apache/solr/handler/UpdateRequestHandler.html[UpdateRequestHandler] |`_UPDATE` |Add, delete and update indexed documents formatted as SolrXML, CSV, SolrJSON or javabin.
-|<<uploading-data-with-index-handlers.adoc#UploadingDatawithIndexHandlers-CSVUpdateConveniencePaths,`/update/csv`>> |{solr-javadocs}/solr-core/org/apache/solr/handler/UpdateRequestHandler.html[UpdateRequestHandler] |`_UPDATE_CSV` |Add and update CSV-formatted documents.
-|<<uploading-data-with-index-handlers.adoc#UploadingDatawithIndexHandlers-CSVUpdateConveniencePaths,`/update/json`>> |{solr-javadocs}/solr-core/org/apache/solr/handler/UpdateRequestHandler.html[UpdateRequestHandler] |`_UPDATE_JSON` |Add, delete and update SolrJSON-formatted documents.
+|<<uploading-data-with-index-handlers.adoc#csv-update-convenience-paths,`/update/csv`>> |{solr-javadocs}/solr-core/org/apache/solr/handler/UpdateRequestHandler.html[UpdateRequestHandler] |`_UPDATE_CSV` |Add and update CSV-formatted documents.
+|<<uploading-data-with-index-handlers.adoc#csv-update-convenience-paths,`/update/json`>> |{solr-javadocs}/solr-core/org/apache/solr/handler/UpdateRequestHandler.html[UpdateRequestHandler] |`_UPDATE_JSON` |Add, delete and update SolrJSON-formatted documents.
|<<transforming-and-indexing-custom-json.adoc#transforming-and-indexing-custom-json,`/update/json/docs`>> |{solr-javadocs}/solr-core/org/apache/solr/handler/UpdateRequestHandler.html[UpdateRequestHandler] |`_UPDATE_JSON_DOCS` |Add and update custom JSON-formatted documents.
|===
-[[ImplicitRequestHandlers-HowtoViewtheConfiguration]]
== How to View the Configuration
You can see configuration for all request handlers, including the implicit request handlers, via the <<config-api.adoc#config-api,Config API>>. E.g. for the `gettingstarted` collection:
@@ -71,7 +69,6 @@ To include the expanded paramset in the response, as well as the effective param
`curl "http://localhost:8983/solr/gettingstarted/config/requestHandler?componentName=/export&expandParams=true"`
-[[ImplicitRequestHandlers-HowtoEdittheConfiguration]]
== How to Edit the Configuration
Because implicit request handlers are not present in `solrconfig.xml`, configuration of their associated `default`, `invariant` and `appends` parameters may be edited via<<request-parameters-api.adoc#request-parameters-api, Request Parameters API>> using the paramset listed in the above table. However, other parameters, including SearchHandler components, may not be modified. The invariants and appends specified in the implicit configuration cannot be overridden.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8d00e53b/solr/solr-ref-guide/src/index-replication.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/index-replication.adoc b/solr/solr-ref-guide/src/index-replication.adoc
index 774b78c..8c51341 100644
--- a/solr/solr-ref-guide/src/index-replication.adoc
+++ b/solr/solr-ref-guide/src/index-replication.adoc
@@ -26,7 +26,6 @@ The figure below shows a Solr configuration using index replication. The master
image::images/index-replication/worddav2b7e14725d898b4104cdd9c502fc77cd.png[image,width=159,height=235]
-[[IndexReplication-IndexReplicationinSolr]]
== Index Replication in Solr
Solr includes a Java implementation of index replication that works over HTTP:
@@ -46,7 +45,6 @@ Although there is no explicit concept of "master/slave" nodes in a <<solrcloud.a
When using SolrCloud, the `ReplicationHandler` must be available via the `/replication` path. Solr does this implicitly unless overridden explicitly in your `solrconfig.xml`, but if you wish to override the default behavior, make certain that you do not explicitly set any of the "master" or "slave" configuration options mentioned below, or they will interfere with normal SolrCloud operation.
====
-[[IndexReplication-ReplicationTerminology]]
== Replication Terminology
The table below defines the key terms associated with Solr replication.
@@ -79,15 +77,13 @@ Snapshot::
A directory containing hard links to the data files of an index. Snapshots are distributed from the master nodes when the slaves pull them, "smart copying" any segments the slave node does not have in snapshot directory that contains the hard links to the most recent index data files.
-[[IndexReplication-ConfiguringtheReplicationHandler]]
== Configuring the ReplicationHandler
In addition to `ReplicationHandler` configuration options specific to the master/slave roles, there are a few special configuration options that are generally supported (even when using SolrCloud).
* `maxNumberOfBackups` an integer value dictating the maximum number of backups this node will keep on disk as it receives `backup` commands.
-* Similar to most other request handlers in Solr you may configure a set of <<requesthandlers-and-searchcomponents-in-solrconfig.adoc#RequestHandlersandSearchComponentsinSolrConfig-SearchHandlers,defaults, invariants, and/or appends>> parameters corresponding with any request parameters supported by the `ReplicationHandler` when <<IndexReplication-HTTPAPICommandsfortheReplicationHandler,processing commands>>.
+* Similar to most other request handlers in Solr you may configure a set of <<requesthandlers-and-searchcomponents-in-solrconfig.adoc#searchhandlers,defaults, invariants, and/or appends>> parameters corresponding with any request parameters supported by the `ReplicationHandler` when <<HTTP API Commands for the ReplicationHandler,processing commands>>.
-[[IndexReplication-ConfiguringtheReplicationRequestHandleronaMasterServer]]
=== Configuring the Replication RequestHandler on a Master Server
Before running a replication, you should set the following parameters on initialization of the handler:
@@ -125,7 +121,6 @@ The example below shows a possible 'master' configuration for the `ReplicationHa
</requestHandler>
----
-[[IndexReplication-Replicatingsolrconfig.xml]]
==== Replicating solrconfig.xml
In the configuration file on the master server, include a line like the following:
@@ -139,7 +134,6 @@ This ensures that the local configuration `solrconfig_slave.xml` will be saved a
On the master server, the file name of the slave configuration file can be anything, as long as the name is correctly identified in the `confFiles` string; then it will be saved as whatever file name appears after the colon ':'.
-[[IndexReplication-ConfiguringtheReplicationRequestHandleronaSlaveServer]]
=== Configuring the Replication RequestHandler on a Slave Server
The code below shows how to configure a ReplicationHandler on a slave.
@@ -188,7 +182,6 @@ The code below shows how to configure a ReplicationHandler on a slave.
</requestHandler>
----
-[[IndexReplication-SettingUpaRepeaterwiththeReplicationHandler]]
== Setting Up a Repeater with the ReplicationHandler
A master may be able to serve only so many slaves without affecting performance. Some organizations have deployed slave servers across multiple data centers. If each slave downloads the index from a remote data center, the resulting download may consume too much network bandwidth. To avoid performance degradation in cases like this, you can configure one or more slaves as repeaters. A repeater is simply a node that acts as both a master and a slave.
@@ -213,7 +206,6 @@ Here is an example of a ReplicationHandler configuration for a repeater:
</requestHandler>
----
-[[IndexReplication-CommitandOptimizeOperations]]
== Commit and Optimize Operations
When a commit or optimize operation is performed on the master, the RequestHandler reads the list of file names which are associated with each commit point. This relies on the `replicateAfter` parameter in the configuration to decide which types of events should trigger replication.
@@ -233,7 +225,6 @@ The `replicateAfter` parameter can accept multiple arguments. For example:
<str name="replicateAfter">optimize</str>
----
-[[IndexReplication-SlaveReplication]]
== Slave Replication
The master is totally unaware of the slaves.
@@ -246,7 +237,6 @@ The slave continuously keeps polling the master (depending on the `pollInterval`
* After the download completes, all the new files are moved to the live index directory and the file's timestamp is same as its counterpart on the master.
* A commit command is issued on the slave by the Slave's ReplicationHandler and the new index is loaded.
-[[IndexReplication-ReplicatingConfigurationFiles]]
=== Replicating Configuration Files
To replicate configuration files, list them using using the `confFiles` parameter. Only files found in the `conf` directory of the master's Solr instance will be replicated.
@@ -259,7 +249,6 @@ As a precaution when replicating configuration files, Solr copies configuration
If a replication involved downloading of at least one configuration file, the ReplicationHandler issues a core-reload command instead of a commit command.
-[[IndexReplication-ResolvingCorruptionIssuesonSlaveServers]]
=== Resolving Corruption Issues on Slave Servers
If documents are added to the slave, then the slave is no longer in sync with its master. However, the slave will not undertake any action to put itself in sync, until the master has new index data.
@@ -268,7 +257,6 @@ When a commit operation takes place on the master, the index version of the mast
To correct this problem, the slave then copies all the index files from master to a new index directory and asks the core to load the fresh index from the new directory.
-[[IndexReplication-HTTPAPICommandsfortheReplicationHandler]]
== HTTP API Commands for the ReplicationHandler
You can use the HTTP commands below to control the ReplicationHandler's operations.
@@ -355,7 +343,6 @@ There are two supported parameters:
* `location`: Location where the snapshot is created.
-[[IndexReplication-DistributionandOptimization]]
== Distribution and Optimization
Optimizing an index is not something most users should generally worry about - but in particular users should be aware of the impacts of optimizing an index when using the `ReplicationHandler`.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8d00e53b/solr/solr-ref-guide/src/index.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/index.adoc b/solr/solr-ref-guide/src/index.adoc
index cc6e7ce..e9200d0 100644
--- a/solr/solr-ref-guide/src/index.adoc
+++ b/solr/solr-ref-guide/src/index.adoc
@@ -1,7 +1,7 @@
= Apache Solr Reference Guide
:page-shortname: index
:page-permalink: index.html
-:page-children: about-this-guide, getting-started, upgrading-solr, using-the-solr-administration-user-interface, documents-fields-and-schema-design, understanding-analyzers-tokenizers-and-filters, indexing-and-basic-data-operations, searching, the-well-configured-solr-instance, managing-solr, solrcloud, legacy-scaling-and-distribution, client-apis, major-changes-from-solr-5-to-solr-6, upgrading-a-solr-cluster, further-assistance, solr-glossary, errata
+:page-children: about-this-guide, getting-started, upgrading-solr, using-the-solr-administration-user-interface, documents-fields-and-schema-design, understanding-analyzers-tokenizers-and-filters, indexing-and-basic-data-operations, searching, the-well-configured-solr-instance, managing-solr, solrcloud, legacy-scaling-and-distribution, client-apis, major-changes-from-solr-5-to-solr-6, upgrading-a-solr-cluster, further-assistance, solr-glossary, errata, how-to-contribute
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8d00e53b/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc b/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc
index 63ab26d..d81936f 100644
--- a/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc
+++ b/solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc
@@ -29,10 +29,8 @@ By default, the settings are commented out in the sample `solrconfig.xml` includ
</indexConfig>
----
-[[IndexConfiginSolrConfig-WritingNewSegments]]
== Writing New Segments
-[[IndexConfiginSolrConfig-ramBufferSizeMB]]
=== ramBufferSizeMB
Once accumulated document updates exceed this much memory space (defined in megabytes), then the pending updates are flushed. This can also create new segments or trigger a merge. Using this setting is generally preferable to `maxBufferedDocs`. If both `maxBufferedDocs` and `ramBufferSizeMB` are set in `solrconfig.xml`, then a flush will occur when either limit is reached. The default is 100Mb.
@@ -42,7 +40,6 @@ Once accumulated document updates exceed this much memory space (defined in mega
<ramBufferSizeMB>100</ramBufferSizeMB>
----
-[[IndexConfiginSolrConfig-maxBufferedDocs]]
=== maxBufferedDocs
Sets the number of document updates to buffer in memory before they are flushed as a new segment. This may also trigger a merge. The default Solr configuration sets to flush by RAM usage (`ramBufferSizeMB`).
@@ -52,20 +49,17 @@ Sets the number of document updates to buffer in memory before they are flushed
<maxBufferedDocs>1000</maxBufferedDocs>
----
-[[IndexConfiginSolrConfig-useCompoundFile]]
=== useCompoundFile
-Controls whether newly written (and not yet merged) index segments should use the <<IndexConfiginSolrConfig-CompoundFileSegments,Compound File Segment>> format. The default is false.
+Controls whether newly written (and not yet merged) index segments should use the <<Compound File Segments>> format. The default is false.
[source,xml]
----
<useCompoundFile>false</useCompoundFile>
----
-[[IndexConfiginSolrConfig-MergingIndexSegments]]
== Merging Index Segments
-[[IndexConfiginSolrConfig-mergePolicyFactory]]
=== mergePolicyFactory
Defines how merging segments is done.
@@ -99,7 +93,6 @@ Choosing the best merge factors is generally a trade-off of indexing speed vs. s
Conversely, keeping more segments can accelerate indexing, because merges happen less often, making an update is less likely to trigger a merge. But searches become more computationally expensive and will likely be slower, because search terms must be looked up in more index segments. Faster index updates also means shorter commit turnaround times, which means more timely search results.
-[[IndexConfiginSolrConfig-CustomizingMergePolicies]]
=== Customizing Merge Policies
If the configuration options for the built-in merge policies do not fully suit your use case, you can customize them: either by creating a custom merge policy factory that you specify in your configuration, or by configuring a {solr-javadocs}/solr-core/org/apache/solr/index/WrapperMergePolicyFactory.html[merge policy wrapper] which uses a `wrapped.prefix` configuration option to control how the factory it wraps will be configured:
@@ -115,9 +108,8 @@ If the configuration options for the built-in merge policies do not fully suit y
</mergePolicyFactory>
----
-The example above shows Solr's {solr-javadocs}/solr-core/org/apache/solr/index/SortingMergePolicyFactory.html[`SortingMergePolicyFactory`] being configured to sort documents in merged segments by `"timestamp desc"`, and wrapped around a `TieredMergePolicyFactory` configured to use the values `maxMergeAtOnce=10` and `segmentsPerTier=10` via the `inner` prefix defined by `SortingMergePolicyFactory` 's `wrapped.prefix` option. For more information on using `SortingMergePolicyFactory`, see <<common-query-parameters.adoc#CommonQueryParameters-ThesegmentTerminateEarlyParameter,the segmentTerminateEarly parameter>>.
+The example above shows Solr's {solr-javadocs}/solr-core/org/apache/solr/index/SortingMergePolicyFactory.html[`SortingMergePolicyFactory`] being configured to sort documents in merged segments by `"timestamp desc"`, and wrapped around a `TieredMergePolicyFactory` configured to use the values `maxMergeAtOnce=10` and `segmentsPerTier=10` via the `inner` prefix defined by `SortingMergePolicyFactory` 's `wrapped.prefix` option. For more information on using `SortingMergePolicyFactory`, see <<common-query-parameters.adoc#segmentterminateearly-parameter,the segmentTerminateEarly parameter>>.
-[[IndexConfiginSolrConfig-mergeScheduler]]
=== mergeScheduler
The merge scheduler controls how merges are performed. The default `ConcurrentMergeScheduler` performs merges in the background using separate threads. The alternative, `SerialMergeScheduler`, does not perform merges with separate threads.
@@ -127,7 +119,6 @@ The merge scheduler controls how merges are performed. The default `ConcurrentMe
<mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler"/>
----
-[[IndexConfiginSolrConfig-mergedSegmentWarmer]]
=== mergedSegmentWarmer
When using Solr in for <<near-real-time-searching.adoc#near-real-time-searching,Near Real Time Searching>> a merged segment warmer can be configured to warm the reader on the newly merged segment, before the merge commits. This is not required for near real-time search, but will reduce search latency on opening a new near real-time reader after a merge completes.
@@ -137,7 +128,6 @@ When using Solr in for <<near-real-time-searching.adoc#near-real-time-searching,
<mergedSegmentWarmer class="org.apache.lucene.index.SimpleMergedSegmentWarmer"/>
----
-[[IndexConfiginSolrConfig-CompoundFileSegments]]
== Compound File Segments
Each Lucene segment is typically comprised of a dozen or so files. Lucene can be configured to bundle all of the files for a segment into a single compound file using a file extension of `.cfs`; it's an abbreviation for Compound File Segment.
@@ -149,16 +139,14 @@ On systems where the number of open files allowed per process is limited, CFS ma
.CFS: New Segments vs Merged Segments
[NOTE]
====
-To configure whether _newly written segments_ should use CFS, see the <<IndexConfiginSolrConfig-useCompoundFile,`useCompoundFile`>> setting described above. To configure whether _merged segments_ use CFS, review the Javadocs for your <<IndexConfiginSolrConfig-mergePolicyFactory,`mergePolicyFactory`>> .
+To configure whether _newly written segments_ should use CFS, see the <<useCompoundFile,`useCompoundFile`>> setting described above. To configure whether _merged segments_ use CFS, review the Javadocs for your <<mergePolicyFactory,`mergePolicyFactory`>> .
-Many <<IndexConfiginSolrConfig-MergingIndexSegments,Merge Policy>> implementations support `noCFSRatio` and `maxCFSSegmentSizeMB` settings with default values that prevent compound files from being used for large segments, but do use compound files for small segments.
+Many <<Merging Index Segments,Merge Policy>> implementations support `noCFSRatio` and `maxCFSSegmentSizeMB` settings with default values that prevent compound files from being used for large segments, but do use compound files for small segments.
====
-[[IndexConfiginSolrConfig-IndexLocks]]
== Index Locks
-[[IndexConfiginSolrConfig-lockType]]
=== lockType
The LockFactory options specify the locking implementation to use.
@@ -177,7 +165,6 @@ For more information on the nuances of each LockFactory, see http://wiki.apache.
<lockType>native</lockType>
----
-[[IndexConfiginSolrConfig-writeLockTimeout]]
=== writeLockTimeout
The maximum time to wait for a write lock on an IndexWriter. The default is 1000, expressed in milliseconds.
@@ -187,7 +174,6 @@ The maximum time to wait for a write lock on an IndexWriter. The default is 1000
<writeLockTimeout>1000</writeLockTimeout>
----
-[[IndexConfiginSolrConfig-OtherIndexingSettings]]
== Other Indexing Settings
There are a few other parameters that may be important to configure for your implementation. These settings affect how or when updates are made to an index.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8d00e53b/solr/solr-ref-guide/src/indexing-and-basic-data-operations.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/indexing-and-basic-data-operations.adoc b/solr/solr-ref-guide/src/indexing-and-basic-data-operations.adoc
index 932ac8e..ece3989 100644
--- a/solr/solr-ref-guide/src/indexing-and-basic-data-operations.adoc
+++ b/solr/solr-ref-guide/src/indexing-and-basic-data-operations.adoc
@@ -43,7 +43,6 @@ This section describes how Solr adds data to its index. It covers the following
* *<<uima-integration.adoc#uima-integration,UIMA Integration>>*: Information about integrating Solr with Apache's Unstructured Information Management Architecture (UIMA). UIMA lets you define custom pipelines of Analysis Engines that incrementally add metadata to your documents as annotations.
-[[IndexingandBasicDataOperations-IndexingUsingClientAPIs]]
== Indexing Using Client APIs
Using client APIs, such as <<using-solrj.adoc#using-solrj,SolrJ>>, from your applications is an important option for updating Solr indexes. See the <<client-apis.adoc#client-apis,Client APIs>> section for more information.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8d00e53b/solr/solr-ref-guide/src/initparams-in-solrconfig.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/initparams-in-solrconfig.adoc b/solr/solr-ref-guide/src/initparams-in-solrconfig.adoc
index 1120e43..180f424 100644
--- a/solr/solr-ref-guide/src/initparams-in-solrconfig.adoc
+++ b/solr/solr-ref-guide/src/initparams-in-solrconfig.adoc
@@ -55,8 +55,7 @@ For example, if an `<initParams>` section has the name "myParams", you can call
[source,xml]
<requestHandler name="/dump1" class="DumpRequestHandler" initParams="myParams"/>
-[[InitParamsinSolrConfig-Wildcards]]
-== Wildcards
+== Wildcards in initParams
An `<initParams>` section can support wildcards to define nested paths that should use the parameters defined. A single asterisk (\*) denotes that a nested path one level deeper should use the parameters. Double asterisks (**) denote all nested paths no matter how deep should use the parameters.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8d00e53b/solr/solr-ref-guide/src/introduction-to-solr-indexing.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/introduction-to-solr-indexing.adoc b/solr/solr-ref-guide/src/introduction-to-solr-indexing.adoc
index 888d8db..83a9378 100644
--- a/solr/solr-ref-guide/src/introduction-to-solr-indexing.adoc
+++ b/solr/solr-ref-guide/src/introduction-to-solr-indexing.adoc
@@ -38,12 +38,10 @@ If the field name is defined in the Schema that is associated with the index, th
For more information on indexing in Solr, see the https://wiki.apache.org/solr/FrontPage[Solr Wiki].
-[[IntroductiontoSolrIndexing-TheSolrExampleDirectory]]
== The Solr Example Directory
When starting Solr with the "-e" option, the `example/` directory will be used as base directory for the example Solr instances that are created. This directory also includes an `example/exampledocs/` subdirectory containing sample documents in a variety of formats that you can use to experiment with indexing into the various examples.
-[[IntroductiontoSolrIndexing-ThecurlUtilityforTransferringFiles]]
== The curl Utility for Transferring Files
Many of the instructions and examples in this section make use of the `curl` utility for transferring content through a URL. `curl` posts and retrieves data over HTTP, FTP, and many other protocols. Most Linux distributions include a copy of `curl`. You'll find curl downloads for Linux, Windows, and many other operating systems at http://curl.haxx.se/download.html. Documentation for `curl` is available here: http://curl.haxx.se/docs/manpage.html.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8d00e53b/solr/solr-ref-guide/src/jvm-settings.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/jvm-settings.adoc b/solr/solr-ref-guide/src/jvm-settings.adoc
index 56560da..532e1a7 100644
--- a/solr/solr-ref-guide/src/jvm-settings.adoc
+++ b/solr/solr-ref-guide/src/jvm-settings.adoc
@@ -24,7 +24,6 @@ Configuring your JVM can be a complex topic and a full discussion is beyond the
For more general information about improving Solr performance, see https://wiki.apache.org/solr/SolrPerformanceFactors.
-[[JVMSettings-ChoosingMemoryHeapSettings]]
== Choosing Memory Heap Settings
The most important JVM configuration settings are those that determine the amount of memory it is allowed to allocate. There are two primary command-line options that set memory limits for the JVM. These are `-Xms`, which sets the initial size of the JVM's memory heap, and `-Xmx`, which sets the maximum size to which the heap is allowed to grow.
@@ -41,12 +40,10 @@ When setting the maximum heap size, be careful not to let the JVM consume all av
On systems with many CPUs/cores, it can also be beneficial to tune the layout of the heap and/or the behavior of the garbage collector. Adjusting the relative sizes of the generational pools in the heap can affect how often GC sweeps occur and whether they run concurrently. Configuring the various settings of how the garbage collector should behave can greatly reduce the overall performance impact when it does run. There is a lot of good information on this topic available on Sun's website. A good place to start is here: http://www.oracle.com/technetwork/java/javase/tech/index-jsp-140228.html[Oracle's Java HotSpot Garbage Collection].
-[[JVMSettings-UsetheServerHotSpotVM]]
== Use the Server HotSpot VM
If you are using Sun's JVM, add the `-server` command-line option when you start Solr. This tells the JVM that it should optimize for a long running, server process. If the Java runtime on your system is a JRE, rather than a full JDK distribution (including `javac` and other development tools), then it is possible that it may not support the `-server` JVM option. Test this by running `java -help` and look for `-server` as an available option in the displayed usage message.
-[[JVMSettings-CheckingJVMSettings]]
== Checking JVM Settings
A great way to see what JVM settings your server is using, along with other useful information, is to use the admin RequestHandler, `solr/admin/system`. This request handler will display a wealth of server statistics and settings.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/8d00e53b/solr/solr-ref-guide/src/kerberos-authentication-plugin.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/kerberos-authentication-plugin.adoc b/solr/solr-ref-guide/src/kerberos-authentication-plugin.adoc
index da96316..a7fa9c1 100644
--- a/solr/solr-ref-guide/src/kerberos-authentication-plugin.adoc
+++ b/solr/solr-ref-guide/src/kerberos-authentication-plugin.adoc
@@ -29,17 +29,14 @@ Support for the Kerberos authentication plugin is available in SolrCloud mode or
If you are using Solr with a Hadoop cluster secured with Kerberos and intend to store your Solr indexes in HDFS, also see the section <<running-solr-on-hdfs.adoc#running-solr-on-hdfs,Running Solr on HDFS>> for additional steps to configure Solr for that purpose. The instructions on this page apply only to scenarios where Solr will be secured with Kerberos. If you only need to store your indexes in a Kerberized HDFS system, please see the other section referenced above.
====
-[[KerberosAuthenticationPlugin-HowSolrWorksWithKerberos]]
== How Solr Works With Kerberos
When setting up Solr to use Kerberos, configurations are put in place for Solr to use a _service principal_, or a Kerberos username, which is registered with the Key Distribution Center (KDC) to authenticate requests. The configurations define the service principal name and the location of the keytab file that contains the credentials.
-[[KerberosAuthenticationPlugin-security.json]]
=== security.json
The Solr authentication model uses a file called `security.json`. A description of this file and how it is created and maintained is covered in the section <<authentication-and-authorization-plugins.adoc#authentication-and-authorization-plugins,Authentication and Authorization Plugins>>. If this file is created after an initial startup of Solr, a restart of each node of the system is required.
-[[KerberosAuthenticationPlugin-ServicePrincipalsandKeytabFiles]]
=== Service Principals and Keytab Files
Each Solr node must have a service principal registered with the Key Distribution Center (KDC). The Kerberos plugin uses SPNego to negotiate authentication.
@@ -56,7 +53,6 @@ Along with the service principal, each Solr node needs a keytab file which shoul
Since a Solr cluster requires internode communication, each node must also be able to make Kerberos enabled requests to other nodes. By default, Solr uses the same service principal and keytab as a 'client principal' for internode communication. You may configure a distinct client principal explicitly, but doing so is not recommended and is not covered in the examples below.
-[[KerberosAuthenticationPlugin-KerberizedZooKeeper]]
=== Kerberized ZooKeeper
When setting up a kerberized SolrCloud cluster, it is recommended to enable Kerberos security for ZooKeeper as well.
@@ -65,15 +61,13 @@ In such a setup, the client principal used to authenticate requests with ZooKeep
See the <<ZooKeeper Configuration>> section below for an example of starting ZooKeeper in Kerberos mode.
-[[KerberosAuthenticationPlugin-BrowserConfiguration]]
=== Browser Configuration
In order for your browser to access the Solr Admin UI after enabling Kerberos authentication, it must be able to negotiate with the Kerberos authenticator service to allow you access. Each browser supports this differently, and some (like Chrome) do not support it at all. If you see 401 errors when trying to access the Solr Admin UI after enabling Kerberos authentication, it's likely your browser has not been configured properly to know how or where to negotiate the authentication request.
Detailed information on how to set up your browser is beyond the scope of this documentation; please see your system administrators for Kerberos for details on how to configure your browser.
-[[KerberosAuthenticationPlugin-PluginConfiguration]]
-== Plugin Configuration
+== Kerberos Authentication Configuration
.Consult Your Kerberos Admins!
[WARNING]
@@ -97,7 +91,6 @@ We'll walk through each of these steps below.
To use host names instead of IP addresses, use the `SOLR_HOST` configuration in `bin/solr.in.sh` or pass a `-Dhost=<hostname>` system parameter during Solr startup. This guide uses IP addresses. If you specify a hostname, replace all the IP addresses in the guide with the Solr hostname as appropriate.
====
-[[KerberosAuthenticationPlugin-GetServicePrincipalsandKeytabs]]
=== Get Service Principals and Keytabs
Before configuring Solr, make sure you have a Kerberos service principal for each Solr host and ZooKeeper (if ZooKeeper has not already been configured) available in the KDC server, and generate a keytab file as shown below.
@@ -128,7 +121,6 @@ Copy the keytab file from the KDC server’s `/tmp/107.keytab` location to the S
You might need to take similar steps to create a ZooKeeper service principal and keytab if it has not already been set up. In that case, the example below shows a different service principal for ZooKeeper, so the above might be repeated with `zookeeper/host1` as the service principal for one of the nodes
-[[KerberosAuthenticationPlugin-ZooKeeperConfiguration]]
=== ZooKeeper Configuration
If you are using a ZooKeeper that has already been configured to use Kerberos, you can skip the ZooKeeper-related steps shown here.
@@ -173,7 +165,6 @@ Once all of the pieces are in place, start ZooKeeper with the following paramete
bin/zkServer.sh start -Djava.security.auth.login.config=/etc/zookeeper/conf/jaas-client.conf
----
-[[KerberosAuthenticationPlugin-Createsecurity.json]]
=== Create security.json
Create the `security.json` file.
@@ -194,7 +185,6 @@ More details on how to use a `/security.json` file in Solr are available in the
If you already have a `/security.json` file in ZooKeeper, download the file, add or modify the authentication section and upload it back to ZooKeeper using the <<command-line-utilities.adoc#command-line-utilities,Command Line Utilities>> available in Solr.
====
-[[KerberosAuthenticationPlugin-DefineaJAASConfigurationFile]]
=== Define a JAAS Configuration File
The JAAS configuration file defines the properties to use for authentication, such as the service principal and the location of the keytab file. Other properties can also be set to ensure ticket caching and other features.
@@ -227,7 +217,6 @@ The main properties we are concerned with are the `keyTab` and `principal` prope
* `debug`: this boolean property will output debug messages for help in troubleshooting.
* `principal`: the name of the service principal to be used.
-[[KerberosAuthenticationPlugin-SolrStartupParameters]]
=== Solr Startup Parameters
While starting up Solr, the following host-specific parameters need to be passed. These parameters can be passed at the command line with the `bin/solr` start command (see <<solr-control-script-reference.adoc#solr-control-script-reference,Solr Control Script Reference>> for details on how to pass system parameters) or defined in `bin/solr.in.sh` or `bin/solr.in.cmd` as appropriate for your operating system.
@@ -252,7 +241,6 @@ The app name (section name) within the JAAS configuration file which is required
`java.security.auth.login.config`::
Path to the JAAS configuration file for configuring a Solr client for internode communication. This parameter is required.
-
Here is an example that could be added to `bin/solr.in.sh`. Make sure to change this example to use the right hostname and the keytab file path.
[source,bash]
@@ -273,7 +261,6 @@ For Java 1.8, this is available here: http://www.oracle.com/technetwork/java/jav
Replace the `local_policy.jar` present in `JAVA_HOME/jre/lib/security/` with the new `local_policy.jar` from the downloaded package and restart the Solr node.
====
-[[KerberosAuthenticationPlugin-UsingDelegationTokens]]
=== Using Delegation Tokens
The Kerberos plugin can be configured to use delegation tokens, which allow an application to reuse the authentication of an end-user or another application.
@@ -304,7 +291,6 @@ The ZooKeeper path where the secret provider information is stored. This is in t
`solr.kerberos.delegation.token.secret.manager.znode.working.path`::
The ZooKeeper path where token information is stored. This is in the form of the path + /security/zkdtsm. The path can include the chroot or the chroot can be omitted if you are not using it. This example includes the chroot: `server1:9983,server2:9983,server3:9983/solr/security/zkdtsm`.
-[[KerberosAuthenticationPlugin-StartSolr]]
=== Start Solr
Once the configuration is complete, you can start Solr with the `bin/solr` script, as in the example below, which is for users in SolrCloud mode only. This example assumes you modified `bin/solr.in.sh` or `bin/solr.in.cmd`, with the proper values, but if you did not, you would pass the system parameters along with the start command. Note you also need to customize the `-z` property as appropriate for the location of your ZooKeeper nodes.
@@ -314,7 +300,6 @@ Once the configuration is complete, you can start Solr with the `bin/solr` scrip
bin/solr -c -z server1:2181,server2:2181,server3:2181/solr
----
-[[KerberosAuthenticationPlugin-TesttheConfiguration]]
=== Test the Configuration
. Do a `kinit` with your username. For example, `kinit \user@EXAMPLE.COM`.
@@ -325,7 +310,6 @@ bin/solr -c -z server1:2181,server2:2181,server3:2181/solr
curl --negotiate -u : "http://192.168.0.107:8983/solr/"
----
-[[KerberosAuthenticationPlugin-UsingSolrJwithaKerberizedSolr]]
== Using SolrJ with a Kerberized Solr
To use Kerberos authentication in a SolrJ application, you need the following two lines before you create a SolrClient:
@@ -353,7 +337,6 @@ SolrJClient {
};
----
-[[KerberosAuthenticationPlugin-DelegationTokenswithSolrJ]]
=== Delegation Tokens with SolrJ
Delegation tokens are also supported with SolrJ, in the following ways: