You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ct...@apache.org on 2017/03/16 17:29:00 UTC
[13/26] lucene-solr:jira/solr-10290: SOLR-10290: Add .adoc files
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/45a148a7/solr/solr-ref-guide/src/overview-of-documents-fields-and-schema-design.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/overview-of-documents-fields-and-schema-design.adoc b/solr/solr-ref-guide/src/overview-of-documents-fields-and-schema-design.adoc
new file mode 100644
index 0000000..cacd60a
--- /dev/null
+++ b/solr/solr-ref-guide/src/overview-of-documents-fields-and-schema-design.adoc
@@ -0,0 +1,56 @@
+= Overview of Documents, Fields, and Schema Design
+:page-shortname: overview-of-documents-fields-and-schema-design
+:page-permalink: overview-of-documents-fields-and-schema-design.html
+
+The fundamental premise of Solr is simple. You give it a lot of information, then later you can ask it questions and find the piece of information you want. The part where you feed in all the information is called _indexing_ or __updating__. When you ask a question, it's called a __query__.
+
+One way to understand how Solr works is to think of a loose-leaf book of recipes. Every time you add a recipe to the book, you update the index at the back. You list each ingredient and the page number of the recipe you just added. Suppose you add one hundred recipes. Using the index, you can very quickly find all the recipes that use garbanzo beans, or artichokes, or coffee, as an ingredient. Using the index is much faster than looking through each recipe one by one. Imagine a book of one thousand recipes, or one million.
+
+Solr allows you to build an index with many different fields, or types of entries. The example above shows how to build an index with just one field, `ingredients`. You could have other fields in the index for the recipe's cooking style, like `Asian`, `Cajun`, or `vegan`, and you could have an index field for preparation times. Solr can answer questions like "What Cajun-style recipes that have blood oranges as an ingredient can be prepared in fewer than 30 minutes?"
+
+The schema is the place where you tell Solr how it should build indexes from input documents.
+
+// OLD_CONFLUENCE_ID: OverviewofDocuments,Fields,andSchemaDesign-HowSolrSeestheWorld
+
+[[OverviewofDocuments_Fields_andSchemaDesign-HowSolrSeestheWorld]]
+== How Solr Sees the World
+
+Solr's basic unit of information is a __document__, which is a set of data that describes something. A recipe document would contain the ingredients, the instructions, the preparation time, the cooking time, the tools needed, and so on. A document about a person, for example, might contain the person's name, biography, favorite color, and shoe size. A document about a book could contain the title, author, year of publication, number of pages, and so on.
+
+In the Solr universe, documents are composed of __fields__, which are more specific pieces of information. Shoe size could be a field. First name and last name could be fields.
+
+Fields can contain different kinds of data. A name field, for example, is text (character data). A shoe size field might be a floating point number so that it could contain values like 6 and 9.5. Obviously, the definition of fields is flexible (you could define a shoe size field as a text field rather than a floating point number, for example), but if you define your fields correctly, Solr will be able to interpret them correctly and your users will get better results when they perform a query.
+
+You can tell Solr about the kind of data a field contains by specifying its __field type__. The field type tells Solr how to interpret the field and how it can be queried.
+
+When you add a document, Solr takes the information in the document's fields and adds that information to an index. When you perform a query, Solr can quickly consult the index and return the matching documents.
+
+// OLD_CONFLUENCE_ID: OverviewofDocuments,Fields,andSchemaDesign-FieldAnalysis
+
+[[OverviewofDocuments_Fields_andSchemaDesign-FieldAnalysis]]
+== Field Analysis
+
+_Field analysis_ tells Solr what to do with incoming data when building an index. A more accurate name for this process would be _processing_ or even __digestion__, but the official name is __analysis__.
+
+Consider, for example, a biography field in a person document. Every word of the biography must be indexed so that you can quickly find people whose lives have had anything to do with ketchup, or dragonflies, or cryptography.
+
+However, a biography will likely contains lots of words you don't care about and don't want clogging up your index\u2014words like "the", "a", "to", and so forth. Furthermore, suppose the biography contains the word "Ketchup", capitalized at the beginning of a sentence. If a user makes a query for "ketchup", you want Solr to tell you about the person even though the biography contains the capitalized word.
+
+The solution to both these problems is field analysis. For the biography field, you can tell Solr how to break apart the biography into words. You can tell Solr that you want to make all the words lower case, and you can tell Solr to remove accents marks.
+
+Field analysis is an important part of a field type. <<understanding-analyzers-tokenizers-and-filters.adoc#understanding-analyzers-tokenizers-and-filters,Understanding Analyzers, Tokenizers, and Filters>> is a detailed description of field analysis.
+
+// OLD_CONFLUENCE_ID: OverviewofDocuments,Fields,andSchemaDesign-Solr'sSchemaFile
+
+[[OverviewofDocuments_Fields_andSchemaDesign-Solr_sSchemaFile]]
+== Solr's Schema File
+
+Solr stores details about the field types and fields it is expected to understand in a schema file. The name and location of this file may vary depending on how you initially configured Solr or if you modified it later.
+
+* `managed-schema` is the name for the schema file Solr uses by default to support making Schema changes at runtime via the <<schema-api.adoc#schema-api,Schema API>>, or <<schemaless-mode.adoc#schemaless-mode,Schemaless Mode>> features. You may <<schema-factory-definition-in-solrconfig.adoc#schema-factory-definition-in-solrconfig,explicitly configure the managed schema features>> to use an alternative filename if you choose, but the contents of the files are still updated automatically by Solr.
+* `schema.xml` is the traditional name for a schema file which can be edited manually by users who use the <<schema-factory-definition-in-solrconfig.adoc#schema-factory-definition-in-solrconfig,`ClassicIndexSchemaFactory`>>.
+* If you are using SolrCloud you may not be able to find any file by these names on the local filesystem. You will only be able to see the schema through the Schema API (if enabled) or through the Solr Admin UI's <<cloud-screens.adoc#cloud-screens,Cloud Screens>>.
+
+Whichever name of the file is being used in your installation, the structure of the file is not changed. However, the way you interact with the file will change. If you are using the managed schema, it is expected that you only interact with the file with the Schema API, and never make manual edits. If you do not use the managed schema, you will only be able to make manual edits to the file, the Schema API will not support any modifications.
+
+Note that if you are not using the Schema API yet you do use SolrCloud, you will need to interact with `schema.xml` through ZooKeeper using upconfig and downconfig commands to make a local copy and upload your changes. The options for doing this are described in <<solr-control-script-reference.adoc#solr-control-script-reference,Solr Control Script Reference>> and <<using-zookeeper-to-manage-configuration-files.adoc#using-zookeeper-to-manage-configuration-files,Using ZooKeeper to Manage Configuration Files>>.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/45a148a7/solr/solr-ref-guide/src/overview-of-searching-in-solr.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/overview-of-searching-in-solr.adoc b/solr/solr-ref-guide/src/overview-of-searching-in-solr.adoc
new file mode 100644
index 0000000..7db8a2c
--- /dev/null
+++ b/solr/solr-ref-guide/src/overview-of-searching-in-solr.adoc
@@ -0,0 +1,47 @@
+= Overview of Searching in Solr
+:page-shortname: overview-of-searching-in-solr
+:page-permalink: overview-of-searching-in-solr.html
+
+Solr offers a rich, flexible set of features for search. To understand the extent of this flexibility, it's helpful to begin with an overview of the steps and components involved in a Solr search.
+
+When a user runs a search in Solr, the search query is processed by a **request handler**. A request handler is a Solr plug-in that defines the logic to be used when Solr processes a request. Solr supports a variety of request handlers. Some are designed for processing search queries, while others manage tasks such as index replication.
+
+Search applications select a particular request handler by default. In addition, applications can be configured to allow users to override the default selection in preference of a different request handler.
+
+To process a search query, a request handler calls a **query parser**, which interprets the terms and parameters of a query. Different query parsers support different syntax. Solr's default query parser is known as the <<the-standard-query-parser.adoc#the-standard-query-parser,Standard Query Parser>>,or more commonly just the "lucene" query parser. Solr also includes the <<the-dismax-query-parser.adoc#the-dismax-query-parser,DisMax>>query parser, and the <<the-extended-dismax-query-parser.adoc#the-extended-dismax-query-parser,Extended DisMax>> (eDisMax) query parser. The <<the-standard-query-parser.adoc#the-standard-query-parser,standard>> query parser's syntax allows for greater precision in searches, but the DisMax query parser is much more tolerant of errors. The DisMax query parser is designed to provide an experience similar to that of popular search engines such as Google, which rarely display syntax errors to users. The Extended DisMax query parser is an improved version of D
isMax that handles the full Lucene query syntax while still tolerating syntax errors. It also includes several additional features.
+
+In addition, there are <<common-query-parameters.adoc#common-query-parameters,common query parameters>> that are accepted by all query parsers.
+
+Input to a query parser can include:
+
+* search strings---that is, _terms_ to search for in the index
+* _parameters for fine-tuning the query_ by increasing the importance of particular strings or fields, by applying Boolean logic among the search terms, or by excluding content from the search results
+* __parameters for controlling the presentation of the query response__, such as specifying the order in which results are to be presented or limiting the response to particular fields of the search application's schema.
+
+Search parameters may also specify a **filter query**. As part of a search response, a filter query runs a query against the entire index and caches the results. Because Solr allocates a separate cache for filter queries, the strategic use of filter queries can improve search performance. (Despite their similar names, query filters are not related to analysis filters. Filter queries perform queries at search time against data already in the index, while analysis filters, such as Tokenizers, parse content for indexing, following specified rules).
+
+A search query can request that certain terms be highlighted in the search response; that is, the selected terms will be displayed in colored boxes so that they "jump out" on the screen of search results. <<highlighting.adoc#highlighting,*Highlighting*>> can make it easier to find relevant passages in long documents returned in a search. Solr supports multi-term highlighting. Solr includes a rich set of search parameters for controlling how terms are highlighted.
+
+Search responses can also be configured to include *snippets* (document excerpts) featuring highlighted text. Popular search engines such as Google and Yahoo! return snippets in their search results: 3-4 lines of text offering a description of a search result.
+
+To help users zero in on the content they're looking for, Solr supports two special ways of grouping search results to aid further exploration: faceting and clustering.
+
+<<faceting.adoc#faceting,*Faceting*>> is the arrangement of search results into categories (which are based on indexed terms). Within each category, Solr reports on the number of hits for relevant term, which is called a facet constraint. Faceting makes it easy for users to explore search results on sites such as movie sites and product review sites, where there are many categories and many items within a category.
+
+The screen shot below shows an example of faceting from the CNET Web site (CBS Interactive Inc.) , which was the first site to use Solr.
+
+image::images/overview-of-searching-in-solr/worddav88969a784fb8a63d8c46e9c043f5f953.png[image,width=600,height=300]
+
+
+Faceting makes use of fields defined when the search applications were indexed. In the example above, these fields include categories of information that are useful for describing digital cameras: manufacturer, resolution, and zoom range.
+
+*Clustering* groups search results by similarities discovered when a search is executed, rather than when content is indexed. The results of clustering often lack the neat hierarchical organization found in faceted search results, but clustering can be useful nonetheless. It can reveal unexpected commonalities among search results, and it can help users rule out content that isn't pertinent to what they're really searching for.
+
+Solr also supports a feature called <<morelikethis.adoc#morelikethis,MoreLikeThis>>, which enables users to submit new queries that focus on particular terms returned in an earlier query. MoreLikeThis queries can make use of faceting or clustering to provide additional aid to users.
+
+A Solr component called a <<response-writers.adoc#response-writers,*response writer*>> manages the final presentation of the query response. Solr includes a variety of response writers, including an <<response-writers.adoc#ResponseWriters-TheStandardXMLResponseWriter,XML Response Writer>> and a <<response-writers.adoc#ResponseWriters-JSONResponseWriter,JSON Response Writer>>.
+
+The diagram below summarizes some key elements of the search process.
+
+image::images/overview-of-searching-in-solr/worddav16392965e726e04513a21641fabad474.png[image,width=624,height=401]
+
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/45a148a7/solr/solr-ref-guide/src/overview-of-the-solr-admin-ui.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/overview-of-the-solr-admin-ui.adoc b/solr/solr-ref-guide/src/overview-of-the-solr-admin-ui.adoc
new file mode 100644
index 0000000..2e529d3
--- /dev/null
+++ b/solr/solr-ref-guide/src/overview-of-the-solr-admin-ui.adoc
@@ -0,0 +1,28 @@
+= Overview of the Solr Admin UI
+:page-shortname: overview-of-the-solr-admin-ui
+:page-permalink: overview-of-the-solr-admin-ui.html
+
+Solr features a Web interface that makes it easy for Solr administrators and programmers to view <<files-screen.adoc#files-screen,Solr configuration>> details, run <<query-screen.adoc#query-screen,queries and analyze>> document fields in order to fine-tune a Solr configuration and access <<getting-assistance.adoc#getting-assistance,online documentation>> and other help.
+
+image::images/overview-of-the-solr-admin-ui/dashboard.png[image,height=400]
+
+
+Accessing the URL `http://hostname:8983/solr/` will show the main dashboard, which is divided into two parts.
+
+A left-side of the screen is a menu under the Solr logo that provides the navigation through the screens of the UI. The first set of links are for system-level information and configuration and provide access to <<logging.adoc#logging,Logging>>,<<collections-core-admin.adoc#collections-core-admin,Collection/Core Administration>> and <<java-properties.adoc#java-properties,Java Properties>>, among other things. At the end of this information is at least one pulldown listing Solr cores configured for this instance. On <<solrcloud.adoc#solrcloud,SolrCloud>> nodes, an additional pulldown list shows all collections in this cluster. Clicking on a collection or core name shows secondary menus of information for the specified collection or core, such as a <<schema-browser-screen.adoc#schema-browser-screen,Schema Browser>>, <<files-screen.adoc#files-screen,Config Files>>, <<plugins-stats-screen.adoc#plugins-stats-screen,Plugins & Statistics>>, and an ability to perform <<query-screen.adoc#que
ry-screen,Queries>> on indexed data.
+
+The center of the screen shows the detail of the option selected. This may include a sub-navigation for the option or text or graphical representation of the requested data. See the sections in this guide for each screen for more details.
+
+Under the covers, the Solr Admin UI re-uses the same HTTP APIs available to all clients to access Solr-related data to drive an external interface.
+
+[TIP]
+====
+
+The path to the Solr Admin UI given above is `http://hostname:port/solr`, which redirects to `http://hostname:port/solr/#/` in the current version. A convenience redirect is also supported, so simply accessing the Admin UI at `http://hostname:port/` will also redirect to `http://hostname:port/solr/#/`.
+
+====
+
+[[OverviewoftheSolrAdminUI-RelatedTopics]]
+== Related Topics
+
+* <<configuring-solrconfig-xml.adoc#configuring-solrconfig-xml,Configuring solrconfig.xml>>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/45a148a7/solr/solr-ref-guide/src/pagination-of-results.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/pagination-of-results.adoc b/solr/solr-ref-guide/src/pagination-of-results.adoc
new file mode 100644
index 0000000..16e57cf
--- /dev/null
+++ b/solr/solr-ref-guide/src/pagination-of-results.adoc
@@ -0,0 +1,273 @@
+= Pagination of Results
+:page-shortname: pagination-of-results
+:page-permalink: pagination-of-results.html
+
+[[PaginationofResults-BasicPagination]]
+== Basic Pagination
+
+In most search application usage, the "top" matching results (sorted by score, or some other criteria) are then displayed to some human user. In many applications the UI for these sorted results are displayed to the user in "pages" containing a fixed number of matching results, and users don't typically look at results past the first few pages worth of results. In Solr, this basic paginated searching is supported using the *`start`* and *`rows`* parameters, and performance of this common behaviour can be tuned by utilizing the <<query-settings-in-solrconfig.adoc#QuerySettingsinSolrConfig-queryResultCache,`queryResultCache`>> and adjusting the <<query-settings-in-solrconfig.adoc#QuerySettingsinSolrConfig-queryResultWindowSize,`queryResultWindowSize`>> configuration options based on your expected page sizes.
+
+[[PaginationofResults-BasicPaginationExamples]]
+=== Basic Pagination Examples
+
+The easiest way to think about simple pagination, is to simply multiply the page number you want (treating the "first" page number as "0") by the number of rows per page; such as in the following psuedo-code:
+
+[source,plain]
+----
+function fetch_solr_page($page_number, $rows_per_page) {
+ $start = $page_number * $rows_per_page
+ $params = [ q = $some_query, rows = $rows_per_page, start = $start ]
+ return fetch_solr($params)
+}
+----
+
+[[PaginationofResults-HowBasicPaginationisAffectedbyIndexUpdates]]
+=== How Basic Pagination is Affected by Index Updates
+
+The `start` param specified in a request to Solr indicates an *absolute* "offset" in the complete sorted list of matches that the client wants Solr to use as the beginning of the current "page". If an index modification (such as adding or removing documents) which affects the sequence of ordered documents matching a query occurs in between two requests from a client for subsequent pages of results, then it is possible that these modifications can result in the same document being returned on multiple pages, or documents being "skipped" as the result set shrinks or grows. For example: consider an index containing 26 documents like so:
+
+id
+
+name
+
+1
+
+A
+
+2
+
+B
+
+...
+
+26
+
+Z
+
+Followed by the following requests & index modifications interleaved:
+
+* A client requests `q=*:*&rows=5&start=0&sort=name asc`
+** documents with the ids `1-5` will be returned to the client
+* Document id `3` is deleted
+* The client requests "page #2" using `q=*:*&rows=5&start=5&sort=name asc`
+** Documents `7-11` will be returned
+** Document `6` has been skipped, since it is now the 5th document in the sorted set of all matching results \u2013 it would be returned on a new request for "page #1"
+* 3 new documents are now added with the ids `90`, `91`, and `92`; All three documents have a name of `A`
+* The client requests "page #3" using `q=*:*&rows=5&start=10&sort=name asc`
+** Documents `9-13` will be returned
+** Documents `9`, `10`, and `11` have now been returned on both page #2 and page #3 since they moved farther back in the list of sorted results
+
+In typical situations these impacts from index changes on paginated searching don't significantly affect user experience -- either because they happen extremely infrequently in fairly static collections, or because the users recognize that the collection of data is constantly evolving and expect to see documents shift up and down in the result sets.
+
+// OLD_CONFLUENCE_ID: PaginationofResults-PerformanceProblemswith"DeepPaging"
+
+[[PaginationofResults-PerformanceProblemswith_DeepPaging_]]
+=== Performance Problems with "Deep Paging"
+
+In some situations, the results of a Solr search are not destined for a simple paginated user interface. When you wish to fetch a very large number of sorted results from Solr to feed into an external system, using very large values for the `start` or `rows` parameters can be very inefficient. Pagination using `start` and `rows` not only require Solr to compute (and sort) in memory all of the matching documents that should be fetched for the current page, but also all of the documents that would have appeared on previous pages. So while a request for `start=0&rows=1000000` may be obviously inefficient because it requires Solr to maintain & sort in memory a set of 1 million documents, likewise a request for `start=999000&rows=1000` is equally inefficient for the same reasons. Solr can't compute which matching document is the 999001st result in sorted order, without first determining what the first 999000 matching sorted results are. If the index is distributed, which is common when r
unning in SolrCloud mode, then 1 million documents are retrieved from **each shard**. For a ten shard index, ten million entries must be retrieved and sorted to figure out the 1000 documents that match those query parameters.
+
+// OLD_CONFLUENCE_ID: PaginationofResults-FetchingALargeNumberofSortedResults:Cursors
+
+[[PaginationofResults-FetchingALargeNumberofSortedResults_Cursors]]
+== Fetching A Large Number of Sorted Results: Cursors
+
+As an alternative to increasing the "start" parameter to request subsequent pages of sorted results, Solr supports using a "Cursor" to scan through results. Cursors in Solr are a logical concept, that doesn't involve caching any state information on the server. Instead the sort values of the last document returned to the client are used to compute a "mark" representing a logical point in the ordered space of sort values. That "mark" can be specified in the parameters of subsequent requests to tell Solr where to continue.
+
+[[PaginationofResults-UsingCursors]]
+=== Using Cursors
+
+To use a cursor with Solr, specify a *`cursorMark`* parameter with the value of `" * "`. You can think of this being analogous to `start=0` as a way to tell Solr "start at the beginning of my sorted results" except that it also informs Solr that you want to use a Cursor. So in addition to returning the top N sorted results (where you can control N using the `rows` parameter) the Solr response will also include an encoded String named *`nextCursorMark`* . You then take the `nextCursorMark` String value from the response, and pass it back to Solr as the `cursorMark` parameter for your next request. You can repeat this process until you've fetched as many docs as you want, or until the `nextCursorMark` returned matches the `cursorMark` you've already specified -- indicating that there are no more results.
+
+[[PaginationofResults-ConstraintswhenusingCursors]]
+=== Constraints when using Cursors
+
+There are a few important constraints to be aware of when using `cursorMark` parameter in a Solr request
+
+// TODO: This 'ol' has problematic nested lists inside of it, needs manual editing
+
+1. `cursorMark` and `start` are mutually exclusive parameters
+* Your requests must either not include a `start` parameter, or it must be specified with a value of "```0```".
+2. `sort` clauses must include the uniqueKey field (either "```asc```" or `"desc`")
+* If `id` is your uniqueKey field, then sort params like `id asc` and `name asc, id desc` would both work fine, but `name asc` by itself would not
+
+Cursor mark values are computed based on the sort values of each document in the result, which means multiple documents with identical sort values will produce identical Cursor mark values if one of them is the last document on a page of results. In that situation, the subsequent request using that `cursorMark` would not know which of the documents with the identical mark values should be skipped. Requiring that the uniqueKey field be used as a clause in the sort criteria guarantees that a deterministic ordering will be returned, and that every `cursorMark` value will identify a unique point in the sequence of documents.
+
+[[PaginationofResults-CursorExamples]]
+=== Cursor Examples
+
+[[PaginationofResults-FetchAllDocs]]
+==== Fetch All Docs
+
+The psuedo-code shown here shows the basic logic involved in fetching all documents matching a query using a cursor:
+
+[source,plain]
+----
+// when fetching all docs, you might as well use a simple id sort
+// unless you really need the docs to come back in a specific order
+$params = [ q => $some_query, sort => 'id asc', rows => $r, cursorMark => '*' ]
+$done = false
+while (not $done) {
+ $results = fetch_solr($params)
+ // do something with $results
+ if ($params[cursorMark] == $results[nextCursorMark]) {
+ $done = true
+ }
+ $params[cursorMark] = $results[nextCursorMark]
+}
+----
+
+Using SolrJ, this psuedo-code would be:
+
+[source,java]
+----
+SolrQuery q = (new SolrQuery(some_query)).setRows(r).setSort(SortClause.asc("id"));
+String cursorMark = CursorMarkParams.CURSOR_MARK_START;
+boolean done = false;
+while (! done) {
+ q.set(CursorMarkParams.CURSOR_MARK_PARAM, cursorMark);
+ QueryResponse rsp = solrServer.query(q);
+ String nextCursorMark = rsp.getNextCursorMark();
+ doCustomProcessingOfResults(rsp);
+ if (cursorMark.equals(nextCursorMark)) {
+ done = true;
+ }
+ cursorMark = nextCursorMark;
+}
+----
+
+If you wanted to do this by hand using curl, the sequence of requests would look something like this:
+
+[source,java]
+----
+$ curl '...&rows=10&sort=id+asc&cursorMark=*'
+{
+ "response":{"numFound":32,"start":0,"docs":[
+ // ... 10 docs here ...
+ ]},
+ "nextCursorMark":"AoEjR0JQ"}
+$ curl '...&rows=10&sort=id+asc&cursorMark=AoEjR0JQ'
+{
+ "response":{"numFound":32,"start":0,"docs":[
+ // ... 10 more docs here ...
+ ]},
+ "nextCursorMark":"AoEpVkRCREIxQTE2"}
+$ curl '...&rows=10&sort=id+asc&cursorMark=AoEpVkRCREIxQTE2'
+{
+ "response":{"numFound":32,"start":0,"docs":[
+ // ... 10 more docs here ...
+ ]},
+ "nextCursorMark":"AoEmbWF4dG9y"}
+$ curl '...&rows=10&sort=id+asc&cursorMark=AoEmbWF4dG9y'
+{
+ "response":{"numFound":32,"start":0,"docs":[
+ // ... 2 docs here because we've reached the end.
+ ]},
+ "nextCursorMark":"AoEpdmlld3Nvbmlj"}
+$ curl '...&rows=10&sort=id+asc&cursorMark=AoEpdmlld3Nvbmlj'
+{
+ "response":{"numFound":32,"start":0,"docs":[
+ // no more docs here, and note that the nextCursorMark
+ // matches the cursorMark param we used
+ ]},
+ "nextCursorMark":"AoEpdmlld3Nvbmlj"}
+----
+
+// OLD_CONFLUENCE_ID: PaginationofResults-FetchfirstNdocs,BasedonPostProcessing
+
+[[PaginationofResults-FetchfirstNdocs_BasedonPostProcessing]]
+==== Fetch first N docs, Based on Post Processing
+
+Since the cursor is stateless from Solr's perspective, your client code can stop fetching additional results as soon as you have decided you have enough information:
+
+[source,java]
+----
+while (! done) {
+ q.set(CursorMarkParams.CURSOR_MARK_PARAM, cursorMark);
+ QueryResponse rsp = solrServer.query(q);
+ String nextCursorMark = rsp.getNextCursorMark();
+ boolean hadEnough = doCustomProcessingOfResults(rsp);
+ if (hadEnough || cursorMark.equals(nextCursorMark)) {
+ done = true;
+ }
+ cursorMark = nextCursorMark;
+}
+----
+
+[[PaginationofResults-HowcursorsareAffectedbyIndexUpdates]]
+=== How cursors are Affected by Index Updates
+
+Unlike basic pagination, Cursor pagination does not rely on using an absolute "offset" into the completed sorted list of matching documents. Instead, the `cursorMark` specified in a request encapsulates information about the *relative* position of the last document returned, based on the *absolute* sort values of that document. This means that the impact of index modifications is much smaller when using a cursor compared to basic pagination. Consider the same example index described when discussing basic pagination:
+
+id
+
+name
+
+1
+
+A
+
+2
+
+B
+
+...
+
+26
+
+Z
+
+* A client requests `q=*:*&rows=5&start=0&sort=name asc, id asc&cursorMark=*`
+** Documents with the ids `1-5` will be returned to the client in order
+* Document id `3` is deleted
+* The client requests 5 more documents using the `nextCursorMark` from the previous response
+** Documents `6-10` will be returned -- the deletion of a document that's already been returned doesn't affect the relative position of the cursor
+* 3 new documents are now added with the ids `90`, `91`, and `92`; All three documents have a name of `A`
+* The client requests 5 more documents using the `nextCursorMark` from the previous response
+** Documents `11-15` will be returned -- the addition of new documents with sort values already past does not affect the relative position of the cursor
+* Document id `1` is updated to change its 'name' to `Q`
+* Document id 17 is updated to change its 'name' to `A`
+* The client requests 5 more documents using the `nextCursorMark` from the previous response
+** The resulting documents are `16,1,18,19,20` in that order
+** Because the sort value of document `1` changed so that it is _after_ the cursor position, the document is returned to the client twice
+** Because the sort value of document `17` changed so that it is _before_ the cursor position, the document has been "skipped" and will not be returned to the client as the cursor continues to progress
+
+In a nutshell: When fetching all results matching a query using `cursorMark`, the only way index modifications can result in a document being skipped, or returned twice, is if the sort value of the document changes.
+
+[TIP]
+====
+
+One way to ensure that a document will never be returned more then once, is to use the uniqueKey field as the primary (and therefore: only significant) sort criterion.
+
+In this situation, you will be guaranteed that each document is only returned once, no matter how it may be be modified during the use of the cursor.
+
+====
+
+// OLD_CONFLUENCE_ID: PaginationofResults-"Tailing"aCursor
+
+[[PaginationofResults-_Tailing_aCursor]]
+=== "Tailing" a Cursor
+
+Because Cursor requests are stateless, and the `cursorMark` values encapsulate the *absolute* sort values of the last document returned from a search, it's possible to "continue" fetching additional results from a cursor that has already reached its end -- if new documents are added (or existing documents are updated) to the end of the results. You can think of this as similar to using something like "`tail -f"` in Unix. The most common examples of how this can be useful is when you have a "timestamp" field recording when a document has been added/updated in your index. Client applications can continuously poll a cursor using a `sort=timestamp asc, id asc` for documents matching a query, and always be notified when a document is added or updated matching the request criteria. Another common example is when you have uniqueKey values that always increase as new documents are created, and you can continuously poll a cursor using `sort=id asc` to be notified about new documents. The psu
edo-code for tailing a cursor is only a slight modification from our early example for processing all docs matching a query:
+
+[source,java]
+----
+while (true) {
+ $doneForNow = false
+ while (not $doneForNow) {
+ $results = fetch_solr($params)
+ // do something with $results
+ if ($params[cursorMark] == $results[nextCursorMark]) {
+ $doneForNow = true
+ }
+ $params[cursorMark] = $results[nextCursorMark]
+ }
+ sleep($some_configured_delay)
+}
+----
+
+[TIP]
+====
+
+For certain specialized cases, the <<exporting-result-sets.adoc#exporting-result-sets,/export handler>> may be an option.
+
+====
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/45a148a7/solr/solr-ref-guide/src/parallel-sql-interface.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/parallel-sql-interface.adoc b/solr/solr-ref-guide/src/parallel-sql-interface.adoc
new file mode 100644
index 0000000..44fdfb3
--- /dev/null
+++ b/solr/solr-ref-guide/src/parallel-sql-interface.adoc
@@ -0,0 +1,482 @@
+= Parallel SQL Interface
+:page-shortname: parallel-sql-interface
+:page-permalink: parallel-sql-interface.html
+:page-children: solr-jdbc-dbvisualizer, solr-jdbc-squirrel-sql, solr-jdbc-apache-zeppelin, solr-jdbc-python-jython, solr-jdbc-r
+
+Solr's Parallel SQL Interface brings the power of SQL to SolrCloud. The SQL interface seamlessly combines SQL with Solr's full-text search capabilities. Two implementations for aggregations allow using either MapReduce-like shuffling or the JSON Facet API, depending on performance needs. These features allow Solr's SQL interface to be used for a wide variety of use cases.
+
+[[ParallelSQLInterface-SQLArchitecture]]
+== SQL Architecture
+
+The SQL interface allows sending a SQL query to Solr and getting documents streamed back in response. Under the covers, Solr's SQL interface is powered by the https://prestodb.io/[Presto Project's] https://github.com/prestodb/presto/tree/master/presto-parser[SQL Parser], which translates SQL queries on the fly to <<streaming-expressions.adoc#streaming-expressions,Streaming Expressions>>.
+
+[[ParallelSQLInterface-SolrCollectionsandDBTables]]
+=== Solr Collections and DB Tables
+
+In a standard `SELECT` statement such as "`SELECT <expressions> FROM <table>`", the table names correspond to Solr collection names. Table names are case insensitive.
+
+Column names in the SQL query map directly to fields in the Solr index for the collection being queried. These identifiers are case sensitive. Aliases are supported, and can be referenced in the `ORDER BY` clause.
+
+The * syntax to indicate all fields is not supported in either limited or unlimited queries. The `score` field can be used only with queries that contain a `LIMIT` clause.
+
+For example, we could index Solr's sample documents and then construct an SQL query like this:
+
+[source,java]
+----
+SELECT manu as mfr, price as retail FROM techproducts
+----
+
+The collection in Solr we are using is "techproducts", and we've asked for the "manu" and "price" fields to be returned and aliased with new names. While this example does not use those aliases, we could build on this to ORDER BY one or more of those fields.
+
+More information about how to structure SQL queries for Solr is included in the section <<ParallelSQLInterface-SolrSQLSyntax,Solr SQL Syntax>>.
+
+[[ParallelSQLInterface-AggregationModes]]
+=== Aggregation Modes
+
+The SQL feature of Solr can work with aggregations (grouping of results) in two ways:
+
+* `map_reduce`: This implementation shuffles tuples to worker nodes and performs the aggregation on the worker nodes. It involves sorting and partitioning the entire result set and sending it to worker nodes. In this approach the tuples arrive at the worker nodes sorted by the GROUP BY fields. The worker nodes can then rollup the aggregates one group at a time. This allows for unlimited cardinality aggregation, but you pay the price of sending the entire result set across the network to worker nodes.
+* `facet`: This uses the JSON Facet API or StatsComponent for aggregations. In this scenario the aggregations logic is pushed down into the search engine and only the aggregates are sent across the network. This is Solr's normal mode of operation. This is fast when the cardinality of GROUP BY fields is low to moderate. But it breaks down when you have high cardinality fields in the GROUP BY field.
+
+These modes are defined with the `aggregationMode` property when sending the request to Solr.
+
+As noted, the choice between aggregation modes depends on the cardinality of the fields you are working with. If you have low-to-moderate cardinality in the fields you are grouping by, the 'facet' aggregation mode will give you a higher performance because only the final groups are returned, very similar to how facets work today. If, however, you have high cardinality in the fields, the "map_reduce" aggregation mode with worker nodes provide a much more performant option.
+
+More detail on the architecture of the "map_reduce" query is in the section <<ParallelSQLInterface-ParallelQueryArchitecture,Parallel Query Architecture>>.
+
+[[ParallelSQLInterface-Configuration]]
+== Configuration
+
+The request handlers used for the SQL interface are configured to load implicitly, meaning there is little to do to start using this feature.
+
+// OLD_CONFLUENCE_ID: ParallelSQLInterface-/sqlRequestHandler
+
+[[ParallelSQLInterface-_sqlRequestHandler]]
+=== /sql Request Handler
+
+The `/sql` handler is the front end of the Parallel SQL interface. All SQL queries are sent to the `/sql` handler to be processed. The handler also coordinates the distributed MapReduce jobs when running `GROUP BY` and `SELECT DISTINCT` queries in `map_reduce` mode. By default the `/sql` handler will choose worker nodes from its own collection to handle the distributed operations. In this default scenario the collection where the `/sql` handler resides acts as the default worker collection for MapReduce queries.
+
+By default, the `/sql` request handler is configured as an implicit handler, meaning that it is always enabled in every Solr installation and no further configuration is required.
+
+[IMPORTANT]
+====
+
+As described below in the section <<ParallelSQLInterface-BestPractices,Best Practices>>, you may want to set up a separate collection for parallelized SQL queries. If you have high cardinality fields and a large amount of data, please be sure to review that section and
+
+====
+
+// OLD_CONFLUENCE_ID: ParallelSQLInterface-/streamand/exportRequestHandlers
+
+[[ParallelSQLInterface-_streamand_exportRequestHandlers]]
+=== /stream and /export Request Handlers
+
+The Streaming API is an extensible parallel computing framework for SolrCloud. <<streaming-expressions.adoc#streaming-expressions,Streaming Expressions>> provide a query language and a serialization format for the Streaming API. The Streaming API provides support for fast MapReduce allowing it to perform parallel relational algebra on extremely large data sets. Under the covers the SQL interface parses SQL queries using the Presto SQL Parser. It then translates the queries to the parallel query plan. The parallel query plan is expressed using the Streaming API and Streaming Expressions.
+
+Like the `/sql` request handler, the `/stream` and `/export` request handlers are configured as implicit handlers, and no further configuration is required.
+
+[[ParallelSQLInterface-Fields]]
+=== Fields
+
+In some cases, fields used in SQL queries must be configured as DocValue fields. If queries are unlimited, all fields must be DocValue fields. If queries are limited (with the `limit` clause) then fields do not have to be have DocValues enabled.
+
+[[ParallelSQLInterface-SendingQueries]]
+== Sending Queries
+
+The SQL Interface provides a basic JDBC driver and an HTTP interface to perform queries.
+
+[[ParallelSQLInterface-JDBCDriver]]
+=== JDBC Driver
+
+The JDBC Driver ships with SolrJ. Below is sample code for creating a connection and executing a query with the JDBC driver:
+
+[source,java]
+----
+Connection con = null;
+try {
+ con = DriverManager.getConnection("jdbc:solr://" + zkHost + "?collection=collection1&aggregationMode=map_reduce&numWorkers=2");
+ stmt = con.createStatement();
+ rs = stmt.executeQuery("SELECT a_s, sum(a_f) as sum FROM collection1 GROUP BY a_s ORDER BY sum desc");
+
+ while(rs.next()) {
+ String a_s = rs.getString("a_s");
+ double s = rs.getDouble("sum");
+ }
+} finally {
+ rs.close();
+ stmt.close();
+ con.close();
+}
+----
+
+The connection URL must contain the `zkHost` and the `collection` parameters. The collection must be a valid SolrCloud collection at the specified ZooKeeper host. The collection must also be configured with the `/sql` handler. The `aggregationMode` and `numWorkers` parameters are optional.
+
+[[ParallelSQLInterface-HTTPInterface]]
+=== HTTP Interface
+
+Solr accepts parallel SQL queries through the `/sql` handler.
+
+Below is a sample curl command performing a SQL aggregate query in facet mode:
+
+[source,bash]
+----
+curl --data-urlencode 'stmt=SELECT to, count(*) FROM collection4 GROUP BY to ORDER BY count(*) desc LIMIT 10'
+ http://localhost:8983/solr/collection4/sql?aggregationMode=facet
+----
+
+Below is sample result set:
+
+[source,java]
+----
+{"result-set":{"docs":[
+ {"count(*)":9158,"to":"pete.davis@enron.com"},
+ {"count(*)":6244,"to":"tana.jones@enron.com"},
+ {"count(*)":5874,"to":"jeff.dasovich@enron.com"},
+ {"count(*)":5867,"to":"sara.shackleton@enron.com"},
+ {"count(*)":5595,"to":"steven.kean@enron.com"},
+ {"count(*)":4904,"to":"vkaminski@aol.com"},
+ {"count(*)":4622,"to":"mark.taylor@enron.com"},
+ {"count(*)":3819,"to":"kay.mann@enron.com"},
+ {"count(*)":3678,"to":"richard.shapiro@enron.com"},
+ {"count(*)":3653,"to":"kate.symes@enron.com"},
+ {"EOF":"true","RESPONSE_TIME":10}]}
+}
+----
+
+Notice that the result set is an array of tuples with key/value pairs that match the SQL column list. The final tuple contains the EOF flag which signals the end of the stream.
+
+[[ParallelSQLInterface-SolrSQLSyntax]]
+== Solr SQL Syntax
+
+Solr supports a broad range of SQL syntax.
+
+.SQL Parser is Case Insensitive
+[IMPORTANT]
+====
+
+The SQL parser being used by Solr to translate the SQL statements is case insensitive. However, for ease of reading, all examples on this page use capitalized keywords.
+
+====
+
+[[ParallelSQLInterface-SELECTStatements]]
+=== SELECT Statements
+
+Solr supports limited and unlimited select queries. The syntax between the two types of queries are identical except for the `LIMIT` clause in the SQL statement. However, they have very different execution plans and different requirements for how the data is stored. The sections below explores both types of queries.
+
+[[ParallelSQLInterface-BasicSELECTstatementwithLIMIT]]
+==== Basic SELECT statement with LIMIT
+
+A limited select query follows this basic syntax:
+
+[source,java]
+----
+SELECT fieldA as fa, fieldB as fb, fieldC as fc FROM tableA WHERE fieldC = 'term1 term2' ORDER BY fa desc LIMIT 100
+----
+
+We've covered many syntax options with this example, so let's walk through what's possible below.
+
+[[ParallelSQLInterface-WHEREClauseandBooleanPredicates]]
+==== *WHERE* Clause *and Boolean Predicates*
+
+[IMPORTANT]
+====
+
+The WHERE clause must have a field on one side of the predicate. Two constants (5 < 10) or two fields (fielda > fieldb) is not supported. Subqueries are also not supported.
+
+====
+
+The `WHERE` clause allows Solr's search syntax to be injected into the SQL query. In the example:
+
+[source,java]
+----
+WHERE fieldC = 'term1 term2'
+----
+
+The predicate above will execute a full text search for the phrase 'term1 term2' in fieldC.
+
+To execute a non-phrase query, simply add parens inside of the single quotes. For example:
+
+[source,java]
+----
+WHERE fieldC = '(term1 term2)'
+----
+
+The predicate above searches for `term1` OR `term2` in `fieldC`.
+
+The Solr range query syntax can be used as follows:
+
+[source,java]
+----
+WHERE fieldC = '[0 TO 100]'
+----
+
+Complex boolean queries can be specified as follows:
+
+[source,java]
+----
+WHERE ((fieldC = 'term1' AND fieldA = 'term2') OR (fieldB = 'term3'))
+----
+
+To specify NOT queries, you use the `AND NOT` syntax as follows:
+
+[source,java]
+----
+WHERE (fieldA = 'term1') AND NOT (fieldB = 'term2')
+----
+
+[[ParallelSQLInterface-SupportedWHEREOperators]]
+==== Supported WHERE *Operators*
+
+The parallel SQL interface supports and pushes down most common SQL operators, specifically:
+
+// TODO: This table has cells that won't work with PDF: https://github.com/ctargett/refguide-asciidoc-poc/issues/13
+
+[width="100%",cols="25%,25%,25%,25%",options="header",]
+|===
+|Operator |Description |Example |Solr Query
+|= |Equals |`fielda = 10` |`fielda:10`
+|<> |Does not equal |`fielda <> 10` |`-fielda:10`
+|!= |Does not equal |`fielda != 10` |`-fielda:10`
+|> |Greater than |`fielda > 10` a|
+....
+fielda:{10 TO *]
+....
+
+|>= |Greater than or equals |`fielda >= 10` a|
+....
+fielda:[10 TO *]
+....
+
+|< |Less than |`fielda < 10` a|
+....
+fielda:[* TO 10}
+....
+
+|<= |Less than or equals |`fielda <= 10` a|
+....
+fielda:[* TO 10]
+....
+
+|===
+
+Some operators that are not supported are BETWEEN, LIKE and IN. However, there are workarounds for BETWEEN and LIKE.
+
+* BETWEEN can be supported with a range query, such as `field = [50 TO 100]`.
+* A simplistic LIKE can be used with a wildcard, such as `field = 'sam*'`.
+
+[[ParallelSQLInterface-ORDERBYClause]]
+==== *ORDER* BY *Clause*
+
+The `ORDER BY` clause maps directly to Solr fields. Multiple `ORDER BY` fields and directions are supported.
+
+The `score` field is accepted in the `ORDER BY` clause in queries where a limit is specified.
+
+Order by fields are case sensitive.
+
+[[ParallelSQLInterface-LIMITClause]]
+==== *LIMIT Clause*
+
+Limits the result set to the specified size. In the example above the clause `LIMIT 100` will limit the result set to 100 records.
+
+There are a few differences to note between limited and unlimited queries:
+
+* Limited queries support `score` in the field list and `ORDER BY`. Unlimited queries do not.
+* Limited queries allow any stored field in the field list. Unlimited queries require the fields to be stored as a DocValues field.
+* Limited queries allow any indexed field in the `ORDER BY` list. Unlimited queries require the fields to be stored as a DocValues field.
+
+[[ParallelSQLInterface-SELECTDISTINCTQueries]]
+==== SELECT DISTINCT Queries
+
+The SQL interface supports both MapReduce and Facet implementations for `SELECT DISTINCT` queries.
+
+The MapReduce implementation shuffles tuples to worker nodes where the Distinct operation is performed. This implementation can perform the Distinct operation over extremely high cardinality fields.
+
+The Facet implementation pushes down the Distinct operation into the search engine using the JSON Facet API. This implementation is designed for high performance, high QPS scenarios on low-to-moderate cardinality fields.
+
+The `aggregationMode` parameter is available in the both the JDBC driver and HTTP interface to choose the underlying implementation (`map_reduce` or `facet`). The SQL syntax is identical for both implementations:
+
+[source,java]
+----
+SELECT distinct fieldA as fa, fieldB as fb FROM tableA ORDER BY fa desc, fb desc
+----
+
+[[ParallelSQLInterface-Statistics]]
+=== Statistics
+
+The SQL interface supports simple statistics calculated on numeric fields. The supported functions are `count(*)`, `min`, `max`, `sum`, and `avg`.
+
+Because these functions never require data to be shuffled, the aggregations are pushed down into the search engine and are generated by the <<the-stats-component.adoc#the-stats-component,StatsComponent>>.
+
+[source,java]
+----
+SELECT count(fieldA) as count, sum(fieldB) as sum FROM tableA WHERE fieldC = 'Hello'
+----
+
+[[ParallelSQLInterface-GROUPBYAggregations]]
+=== GROUP BY Aggregations
+
+The SQL interface also supports `GROUP BY` aggregate queries.
+
+As with `SELECT DISTINCT` queries, the SQL interface supports both a MapReduce implementation and a Facet implementation. The MapReduce implementation can build aggregations over extremely high cardinality fields. The Facet implementations provides high performance aggregation over fields with moderate levels of cardinality.
+
+[[ParallelSQLInterface-BasicGROUPBYwithAggregates]]
+==== *Basic `GROUP BY` with Aggregates*
+
+Here is a basic example of a GROUP BY query that requests aggregations:
+
+[source,java]
+----
+SELECT fieldA as fa, fieldB as fb, count(*) as count, sum(fieldC) as sum, avg(fieldY) as avg FROM tableA WHERE fieldC = 'term1 term2'
+GROUP BY fa, fb HAVING sum > 1000 ORDER BY sum asc LIMIT 100
+----
+
+Let's break this down into pieces:
+
+[[ParallelSQLInterface-TheColumnIdentifiersandAliases]]
+==== *The Column Identifiers and Aliases*
+
+The Column Identifiers can contain both fields in the Solr index and aggregate functions. The supported aggregate functions are:
+
+* `count(*)`: Counts the number of records over a set of buckets.
+* `sum(field)`: Sums a numeric field over over a set of buckets.
+* `avg(field)`: Averages a numeric field over a set of buckets.
+* `min(field)`: Returns the min value of a numeric field over a set of buckets.
+* `max:(field)`: Returns the max value of a numerics over a set of buckets.
+
+The non-function fields in the field list determine the fields to calculate the aggregations over.
+
+Column aliases are supported for both fields and functions and can be referenced in the GROUP BY, HAVING and ORDER BY clauses.
+
+[[ParallelSQLInterface-GROUPBYClause]]
+==== *GROUP BY Clause*
+
+The `GROUP BY` clause can contain up to 4 fields in the Solr index. These fields should correspond with the non-function fields in the field list.
+
+[[ParallelSQLInterface-HAVINGClause]]
+==== *HAVING Clause*
+
+The `HAVING` clause may contain any function listed in the field list. Complex `HAVING` clauses such as this are supported:
+
+[source,java]
+----
+SELECT fieldA, fieldB, count(*), sum(fieldC), avg(fieldY)
+FROM tableA
+WHERE fieldC = 'term1 term2'
+GROUP BY fieldA, fieldB
+HAVING ((sum(fieldC) > 1000) AND (avg(fieldY) <= 10))
+ORDER BY sum(fieldC) asc
+LIMIT 100
+----
+
+[[ParallelSQLInterface-ORDERBYClause.1]]
+==== *ORDER BY Clause*
+
+The `ORDER BY` clause contains any field or function in the field list.
+
+If the `ORDER BY` clause contains the exact fields in the `GROUP BY` clause, then there is no-limit placed on the returned results. If the `ORDER BY` clause contains different fields than the `GROUP BY` clause, a limit of 100 is automatically applied. To increase this limit you must specify a value in the `LIMIT` clause.
+
+[[ParallelSQLInterface-BestPractices]]
+== Best Practices
+
+[[ParallelSQLInterface-SeparateCollections]]
+=== Separate Collections
+
+It makes sense to create a separate SolrCloud collection just for the `/sql` handler. This collection can be created using SolrCloud's standard collection API. Since this collection only exists to handle `/sql` requests and provide a pool of worker nodes, this collection does not need to hold any data. Worker nodes are selected randomly from the entire pool of available nodes in the `/sql` handler's collection. So to grow this collection dynamically replicas can be added to existing shards. New replicas will automatically be put to work after they've been added.
+
+[[ParallelSQLInterface-ParallelSQLQueries]]
+== Parallel SQL Queries
+
+An earlier section describes how the SQL interface translates the SQL statement to a streaming expression. One of the parameters of the request is the `aggregationMode`, which defines if the query should use a MapReduce-like shuffling technique or push the operation down into the search engine.
+
+[[ParallelSQLInterface-ParallelizedQueries]]
+=== Parallelized Queries
+
+The Parallel SQL architecture consists of three logical tiers: a *SQL* tier, a *Worker* tier, and a *Data Table* tier. By default the SQL and Worker tiers are collapsed into the same physical SolrCloud collection.
+
+[[ParallelSQLInterface-SQLTier]]
+==== SQL Tier
+
+The SQL tier is where the `/sql` handler resides. The `/sql` handler takes the SQL query and translates it to a parallel query plan. It then selects worker nodes to execute the plan and sends the query plan to each worker node to be run in parallel.
+
+Once the query plan has been executed by the worker nodes, the `/sql` handler then performs the final merge of the tuples returned by the worker nodes.
+
+[[ParallelSQLInterface-WorkerTier]]
+==== Worker Tier
+
+The workers in the worker tier receive the query plan from the `/sql` handler and execute the parallel query plan. The parallel execution plan includes the queries that need to be made on the Data Table tier and the relational algebra needed to satisfy the query. Each worker node assigned to the query is shuffled 1/N of the tuples from the Data Tables. The worker nodes execute the query plan and stream tuples back to the worker nodes.
+
+[[ParallelSQLInterface-DataTableTier]]
+==== Data Table Tier
+
+The Data Table tier is where the tables reside. Each table is its own SolrCloud collection. The Data Table layer receives queries from the worker nodes and emits tuples (search results). The Data Table tier also handles the initial sorting and partitioning of tuples sent to the workers. This means the tuples are always sorted and partitioned before they hit the network. The partitioned tuples are sent directly to the correct worker nodes in the proper sort order, ready to be reduced.
+
+image::images/parallel-sql-interface/cluster.png[image,width=492,height=250]
+
+
+The image above shows the three tiers broken out into different SolrCloud collections for clarity. In practice the `/sql` handler and worker collection by default share the same collection.
+
+*Note:* The image shows the network flow for a single Parallel SQL Query (SQL over MapReduce). This network flow is used when `map_reduce` aggregation mode is used for `GROUP BY` aggregations or the `SELECT DISTINCT` query. The traditional SolrCloud network flow (without workers) is used when the `facet` aggregation mode is used.
+
+Below is a description of the flow:
+
+1. The client sends a SQL query to the `/sql` handler. The request is handled by a single `/sql` handler instance.
+2. The `/sql` handler parses the SQL query and creates the parallel query plan.
+3. The query plan is sent to worker nodes (in green).
+4. The worker nodes execute the plan in parallel. The diagram shows each worker node contacting a collection in the Data Table tier (in blue).
+5. The collection in the Data Table tier is the table from the SQL query. Notice that the collection has five shards each with 3 replicas.
+6. Notice that each worker contacts one replica from each shard. Because there are 5 workers, each worker is returned 1/5 of the search results from each shard. The partitioning is done inside of the Data Table tier so there is no duplication of data across the network.
+7. Also notice with this design ALL replicas in the data layer are shuffling (sorting & partitioning) data simultaneously. As the number of shards, replicas and workers grows this design allows for a massive amount of computing power to be applied to a single query.
+8. The worker nodes process the tuples returned from the Data Table tier in parallel. The worker nodes perform the relational algebra needed to satisfy the query plan.
+9. The worker nodes stream tuples back to the `/sql` handler where the final merge is done, and finally the tuples are streamed back to the client.
+
+[[ParallelSQLInterface-SQLClientsandDatabaseVisualizationTools]]
+== SQL Clients and Database Visualization Tools
+
+The SQL interface supports queries sent from SQL clients and database visualization tools such as DbVisualizer and Apache Zeppelin.
+
+[[ParallelSQLInterface-Generic]]
+=== Generic
+
+For most Java based clients, the following jars will need to be placed on the client classpath:
+
+* all .jars found in `$SOLR_HOME/dist/solrj-libs`
+* the SolrJ .jar found at `$SOLR_HOME/dist/solr-solrj-<version>.jar`
+
+If you are using Maven, the `org.apache.solr.solr-solrj` artifact contains the required jars.
+
+Once the jars are available on the classpath, the Solr JDBC driver name is `org.apache.solr.client.solrj.io.sql.DriverImpl` and a connection can be made with the following connection string format:
+
+[source,java]
+----
+jdbc:solr://SOLR_ZK_CONNECTION_STRING?collection=COLLECTION_NAME
+----
+
+There are other parameters that can be optionally added to the connection string like `aggregationMode` and `numWorkers`. An example of a Java connection is available in the section <<ParallelSQLInterface-JDBCDriver,JDBC Driver>>.
+
+[[ParallelSQLInterface-DbVisualizer]]
+=== DbVisualizer
+
+A step-by-step guide for setting up https://www.dbvis.com/[DbVisualizer] is in the section <<solr-jdbc-dbvisualizer.adoc#solr-jdbc-dbvisualizer,Solr JDBC - DbVisualizer>>.
+
+[[ParallelSQLInterface-SQuirreLSQL]]
+=== SQuirreL SQL
+
+A step-by-step guide for setting up http://squirrel-sql.sourceforge.net[SQuirreL SQL] is in the section <<solr-jdbc-squirrel-sql.adoc#solr-jdbc-squirrel-sql,Solr JDBC - SQuirreL SQL>>.
+
+// OLD_CONFLUENCE_ID: ParallelSQLInterface-ApacheZeppelin(incubating)
+
+[[ParallelSQLInterface-ApacheZeppelin_incubating_]]
+=== Apache Zeppelin (incubating)
+
+A step-by-step guide for setting up http://zeppelin.apache.org/[Apache Zeppelin] is in the section <<solr-jdbc-apache-zeppelin.adoc#solr-jdbc-apache-zeppelin,Solr JDBC - Apache Zeppelin>>.
+
+// OLD_CONFLUENCE_ID: ParallelSQLInterface-Python/Jython
+
+[[ParallelSQLInterface-Python_Jython]]
+=== Python/Jython
+
+Examples of using Python and Jython for connecting to Solr with the Solr JDBC driver are available in the section <<solr-jdbc-python-jython.adoc#solr-jdbc-python-jython,Solr JDBC - Python/Jython>>.
+
+[[ParallelSQLInterface-R]]
+=== R
+
+Examples of using R for connecting to Solr with the Solr JDBC driver are available in the section <<solr-jdbc-r.adoc#solr-jdbc-r,Solr JDBC - R>> .
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/45a148a7/solr/solr-ref-guide/src/parameter-reference.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/parameter-reference.adoc b/solr/solr-ref-guide/src/parameter-reference.adoc
new file mode 100644
index 0000000..b8212d5
--- /dev/null
+++ b/solr/solr-ref-guide/src/parameter-reference.adoc
@@ -0,0 +1,47 @@
+= Parameter Reference
+:page-shortname: parameter-reference
+:page-permalink: parameter-reference.html
+
+[[ParameterReference-ClusterParameters]]
+=== Cluster Parameters
+
+[width="100%",cols="34%,33%,33%",]
+|===
+|`numShards` |Defaults to 1 |The number of shards to hash documents to. There must be one leader per shard and each leader can have N replicas.
+|===
+
+[[ParameterReference-SolrCloudInstanceParameters]]
+=== SolrCloud Instance Parameters
+
+These are set in `solr.xml`, but by default the `host` and `hostContext` parameters are set up to also work with system properties.
+
+[width="100%",cols="34%,33%,33%",]
+|===
+|`host` |Defaults to the first local host address found |If the wrong host address is found automatically, you can override the host address with this parameter.
+|`hostPort` |Defaults to the port specified via `bin/solr -p <port>`, or `8983` if not specified. |The port that Solr is running on. This value is only used when `-DzkRun` is specified without a value (see below), to calculate the default port on which embedded ZooKeeper will run. **I**n the `solr.xml` shipped with Solr, the `hostPort` system property is not referenced, and so is ignored. If you want to run Solr on a non-default port, use `bin/solr -p <port>` rather than specifying `-DhostPort`.
+|`hostContext` |Defaults to `solr` |The context path for the Solr web application.
+|===
+
+[[ParameterReference-SolrCloudInstanceZooKeeperParameters]]
+=== SolrCloud Instance ZooKeeper Parameters
+
+[width="100%",cols="34%,33%,33%",]
+|===
+|`zkRun` |Defaults to `localhost:<hostPort+1000>` |Causes Solr to run an embedded version of ZooKeeper. Set to the address of ZooKeeper on this node; this allows us to know who you are in the list of addresses in the `zkHost` connect string. Use `-DzkRun` (with no value) to get the default value.
+|`zkHost` |No default |The host address for ZooKeeper. Usually this is a comma-separated list of addresses to each node in your ZooKeeper ensemble.
+|`zkClientTimeout` |Defaults to 15000 |The time a client is allowed to not talk to ZooKeeper before its session expires.
+|===
+
+`zkRun` and `zkHost` are set up using system properties. `zkClientTimeout` is set up in `solr.xml` by default, but can also be set using a system property.
+
+[[ParameterReference-SolrCloudCoreParameters]]
+=== SolrCloud Core Parameters
+
+[width="100%",cols="34%,33%,33%",]
+|===
+|`shard` |Defaults to being automatically assigned based on numShards |Specifies which shard this core acts as a replica of.
+|===
+
+`shard` can be specified in the <<defining-core-properties.adoc#defining-core-properties,`core.properties`>> for each core.
+
+Additional cloud related parameters are discussed in <<format-of-solr-xml.adoc#format-of-solr-xml,Format of solr.xml>>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/45a148a7/solr/solr-ref-guide/src/performance-statistics-reference.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/performance-statistics-reference.adoc b/solr/solr-ref-guide/src/performance-statistics-reference.adoc
new file mode 100644
index 0000000..97cc4cd
--- /dev/null
+++ b/solr/solr-ref-guide/src/performance-statistics-reference.adoc
@@ -0,0 +1,120 @@
+= Performance Statistics Reference
+:page-shortname: performance-statistics-reference
+:page-permalink: performance-statistics-reference.html
+
+This page explains some of the <<using-jmx-with-solr.adoc#using-jmx-with-solr,JMX>> statistics that Solr exposes. The same statistics are also exposed via the<<mbean-request-handler.adoc#mbean-request-handler,MBean Request Handler>> when statistics are requested.
+
+These statistics are per core. When you are running in SolrCloud mode these statistics would co-relate to each performance of an individual replica.
+
+[[PerformanceStatisticsReference-RequestHandlers]]
+== Request Handlers
+
+[[PerformanceStatisticsReference-UpdateRequestHandler]]
+=== Update Request Handler
+
+The update request handler is an endpoint to send data to Solr. We can see how many update requests are being fired, how fast is it performing, and other valuable information regarding requests.
+
+*Path:* `/solr/<core>/update`
+
+[[PerformanceStatisticsReference-SearchRequestHandler]]
+=== Search Request Handler
+
+Can be useful to measure and track number of search queries , response times etc. If you are not using the \u201cselect\u201d handler then the path needs to be changed appropriately. Similarly if you are using the \u201csql\u201d handler or \u201cexport\u201d handler , the realtime handler \u201cget\u201d, or any other handler similar statistics can be found for that as well.
+
+**Path**: `/solr/<core>/select`
+
+Both Update Request Handler and Search Request Handler along with handlers like \u201csql\u201d, \u201cexport\u201d and realtime \u201cget\u201d handler will provide the following attributes in their statistics.
+
+[width="100%",cols="50%,50%",options="header",]
+|===
+|Attribute |Description
+|15minRateReqsPerSecond |Requests per second received over the past 15 minutes.
+|5minRateReqsPerSecond |Requests per second received over the past 5 minutes.
+|75thPcRequestTime |Request processing time for the request which belongs to the 75th Percentile. E.g. if 100 requests are received, then the 75th fastest request time will be reported by this statistic.
+|95thPcRequestTime |Request processing time in milliseconds for the request which belongs to the 95th Percentile. E.g. if 80 requests are received, then the 76th fastest request time will be reported in this statistic.
+|999thPcRequestTime |Request processing time in milliseconds for the request which belongs to the 99.9th Percentile. E.g., if 1000 requests are received, then the 999th fastest request time will be reported in this statistic.
+|99thPcRequestTime |Request processing time in milliseconds for the request which belongs to the 99th Percentile. E.g. if 200 requests are received, then the 198th fastest request time will be reported in this statistic.
+|avgRequestsPerSecond |Average number of requests received per second.
+|avgTimePerRequest |Average time taken for processing the requests.
+|errors |Number of error encountered by handler.
+|clientErrors |Number of syntax errors/parse errors made by client while making requests.
+|handlerStart |Epoch time when the handler was registered.
+|medianRequestTime |Median of all the request processing time.
+|requests |Total number of requests made since the Solr process was started.
+|serverErrors |Number of errors thrown by the server while executing the request.
+|timeouts |Number of responses received with partial results.
+|totalTime |The sum of all request processing times since the Solr process was started.
+|===
+
+[[PerformanceStatisticsReference-UpdateHandler]]
+== Update Handler
+
+*Update Handler:* This section has information on the total number of adds, how many commits have been fired against a solr core.
+
+*Path:* `/solr/<core>/updateHandler/DirectUpdateHandler2`
+
+[width="100%",cols="50%,50%",options="header",]
+|===
+|Attribute |Description
+|adds |Total number of \u201cadd\u201d requests since last commit.
+|autocommit maxTime |Maximum time between two autocommits execution.
+|autocommits |Total number of auto-commits executed.
+|cumulative_adds |Number of \u201ceffective\u201d additions executed over the lifetime. The counter is incremented when \u201cadd\u2018 command is executed while decremented when \u201crollback\u201d is executed.
+|cumulative_deletesById |Number of document deletions executed by ID over the lifetime. The counter is incremented when \u201cdelete\u201d command is executed while decremented when \u201crollback\u201d is executed.
+|cumulative_deletesByQuery |Number of document deletions executed by query over the lifetime. The counter is incremented when \u201cdelete\u201d command is executed while decremented when \u201crollback\u201d is executed.
+|cumulative_errors |Number of error messages received by Update Handler while performing addition/deletion action on documents over the lifetime.
+|deletesById |Currently uncommitted deletions by ID.
+|deletesByQuery |Currently uncommitted deletions by query.
+|docsPending |Number of documents which are pending commit.
+|errors |Number of error messages received by Update Handler while performing addition/deletion/commit/rollback action on documents over the lifetime.
+|expungeDeletes |Number of commit commands issued with expunge deletes.
+|optimizes |Number of explicit optimize commands issued
+|rollbacks |Number of rollbacks executed.
+|soft autocommit maxTime |Maximum documents \u2018adds\u2019 between two soft auto-commits.
+|soft autocommits |Number of soft commits executed.
+|transaction_logs_total_number |Number of TLogs created from the beginning of the Solr instance. It will be equivalent to number of Hard commits executed.
+|transaction_logs_total_size |Total size of all the TLogs created so far from the beginning of the Solr instance.
+|===
+
+[[PerformanceStatisticsReference-Caches]]
+== Caches
+
+[[PerformanceStatisticsReference-DocumentCache]]
+=== Document Cache
+
+This cache holds Lucene Document objects (the stored fields for each document). Since Lucene internal document IDs are transient, this cache cannot be auto-warmed.
+
+*Path:* `/solr/<cache>/documentCache`
+
+[[PerformanceStatisticsReference-QueryResultCache]]
+=== Query Result Cache
+
+This cache holds the results of previous searches: ordered lists of document IDs based on a query, a sort, and the range of documents requested
+
+*Path:* `/solr/<cache>/queryResultCache`
+
+[[PerformanceStatisticsReference-FilterCache]]
+=== Filter Cache
+
+This cache is used for filters for unordered sets of all documents that match a query.
+
+*Path:* `/solr/<cache>/filterCache`
+
+[width="100%",cols="50%,50%",options="header",]
+|===
+|Attribute |Description
+|cumulative_evictions |Number of cache evictions across all caches since this node has been running.
+|cumulative_hitratio |Ratio of cache hits to lookups across all the caches since this node has been running.
+|cumulative_hits |Number of cache hits across all the caches since this node has been running.
+|cumulative_inserts |Number of cache insertions across all the caches since this node has been running.
+|cumulative_lookups |Number of cache lookups across all the caches since this node has been running.
+|evictions |Number of cache evictions for the current index searcher.
+|hitratio |Ratio of cache hits to lookups for the current index searcher.
+|hits |Number of hits for the current index searcher.
+|inserts |Number of inserts into the cache.
+|lookups |Number of lookups against the cache.
+|size |Size of the cache at that particular instance (in KBs).
+|warmupTime |Warm-up time for the registered index searcher. This time is taken in account for the \u201cauto-warming\u201d of caches.
+|===
+
+More information on Solr caches is available in the section <<query-settings-in-solrconfig.adoc#query-settings-in-solrconfig,Query Settings in SolrConfig>>.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/45a148a7/solr/solr-ref-guide/src/phonetic-matching.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/phonetic-matching.adoc b/solr/solr-ref-guide/src/phonetic-matching.adoc
new file mode 100644
index 0000000..a266cda
--- /dev/null
+++ b/solr/solr-ref-guide/src/phonetic-matching.adoc
@@ -0,0 +1,124 @@
+= Phonetic Matching
+:page-shortname: phonetic-matching
+:page-permalink: phonetic-matching.html
+
+Phonetic matching algorithms may be used to encode tokens so that two different spellings that are pronounced similarly will match.
+
+For overviews of and comparisons between algorithms, see http://en.wikipedia.org/wiki/Phonetic_algorithm and http://ntz-develop.blogspot.com/2011/03/phonetic-algorithms.html
+
+// OLD_CONFLUENCE_ID: PhoneticMatching-Beider-MorsePhoneticMatching(BMPM)
+
+[[PhoneticMatching-Beider-MorsePhoneticMatching_BMPM_]]
+== Beider-Morse Phonetic Matching (BMPM)
+
+For examples of how to use this encoding in your analyzer, see <<filter-descriptions.adoc#FilterDescriptions-Beider-MorseFilter,Beider Morse Filter>> in the Filter Descriptions section.
+
+Beider-Morse Phonetic Matching (BMPM) is a "soundalike" tool that lets you search using a new phonetic matching system. BMPM helps you search for personal names (or just surnames) in a Solr/Lucene index, and is far superior to the existing phonetic codecs, such as regular soundex, metaphone, caverphone, etc.
+
+In general, phonetic matching lets you search a name list for names that are phonetically equivalent to the desired name. BMPM is similar to a soundex search in that an exact spelling is not required. Unlike soundex, it does not generate a large quantity of false hits.
+
+From the spelling of the name, BMPM attempts to determine the language. It then applies phonetic rules for that particular language to transliterate the name into a phonetic alphabet. If it is not possible to determine the language with a fair degree of certainty, it uses generic phonetic instead. Finally, it applies language-independent rules regarding such things as voiced and unvoiced consonants and vowels to further insure the reliability of the matches.
+
+For example, assume that the matches found when searching for Stephen in a database are "Stefan", "Steph", "Stephen", "Steve", "Steven", "Stove", and "Stuffin". "Stefan", "Stephen", and "Steven" are probably relevant, and are names that you want to see. "Stuffin", however, is probably not relevant. Also rejected were "Steph", "Steve", and "Stove". Of those, "Stove" is probably not one that we would have wanted. But "Steph" and "Steve" are possibly ones that you might be interested in.
+
+For Solr, BMPM searching is available for the following languages:
+
+* English
+* French
+* German
+* Greek
+* Hebrew written in Hebrew letters
+* Hungarian
+* Italian
+
+* Polish
+* Romanian
+* Russian written in Cyrillic letters
+* Russian transliterated into English letters
+* Spanish
+* Turkish
+
+The name matching is also applicable to non-Jewish surnames from the countries in which those languages are spoken.
+
+For more information, see here: http://stevemorse.org/phoneticinfo.htm and http://stevemorse.org/phonetics/bmpm.htm[http://stevemorse.org/phonetics/bmpm.htm.]
+
+[[PhoneticMatching-Daitch-MokotoffSoundex]]
+== Daitch-Mokotoff Soundex
+
+To use this encoding in your analyzer, see <<filter-descriptions.adoc#FilterDescriptions-Daitch-MokotoffSoundexFilter,Daitch-Mokotoff Soundex Filter>> in the Filter Descriptions section.
+
+The Daitch-Mokotoff Soundex algorithm is a refinement of the Russel and American Soundex algorithms, yielding greater accuracy in matching especially Slavic and Yiddish surnames with similar pronunciation but differences in spelling.
+
+The main differences compared to the other soundex variants are:
+
+* coded names are 6 digits long
+* initial character of the name is coded
+* rules to encoded multi-character n-grams
+* multiple possible encodings for the same name (branching)
+
+Note: the implementation used by Solr (commons-codec's http://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/language/DaitchMokotoffSoundex.html[`DaitchMokotoffSoundex`] ) has additional branching rules compared to the original description of the algorithm.
+
+For more information, see http://en.wikipedia.org/wiki/Daitch%E2%80%93Mokotoff_Soundex and http://www.avotaynu.com/soundex.htm
+
+[[PhoneticMatching-DoubleMetaphone]]
+== Double Metaphone
+
+To use this encoding in your analyzer, see <<filter-descriptions.adoc#FilterDescriptions-DoubleMetaphoneFilter,Double Metaphone Filter>> in the Filter Descriptions section. Alternatively, you may specify `encoding="DoubleMetaphone"` with the <<filter-descriptions.adoc#FilterDescriptions-PhoneticFilter,Phonetic Filter>>, but note that the Phonetic Filter version will *not* provide the second ("alternate") encoding that is generated by the Double Metaphone Filter for some tokens.
+
+Encodes tokens using the double metaphone algorithm by Lawrence Philips. See the original article at http://www.drdobbs.com/the-double-metaphone-search-algorithm/184401251?pgno=2
+
+[[PhoneticMatching-Metaphone]]
+== Metaphone
+
+To use this encoding in your analyzer, specify `encoding="Metaphone"` with the <<filter-descriptions.adoc#FilterDescriptions-PhoneticFilter,Phonetic Filter>>.
+
+Encodes tokens using the Metaphone algorithm by Lawrence Philips, described in "Hanging on the Metaphone" in Computer Language, Dec. 1990.
+
+See http://en.wikipedia.org/wiki/Metaphone
+
+[[PhoneticMatching-Soundex]]
+== Soundex
+
+To use this encoding in your analyzer, specify `encoding="Soundex"` with the <<filter-descriptions.adoc#FilterDescriptions-PhoneticFilter,Phonetic Filter>>.
+
+Encodes tokens using the Soundex algorithm, which is used to relate similar names, but can also be used as a general purpose scheme to find words with similar phonemes.
+
+http://www.drdobbs.com/the-double-metaphone-search-algorithm/184401251?pgno=2[See] http://en.wikipedia.org/wiki/Soundex
+
+[[PhoneticMatching-RefinedSoundex]]
+== Refined Soundex
+
+To use this encoding in your analyzer, specify `encoding="RefinedSoundex"` with the <<filter-descriptions.adoc#FilterDescriptions-PhoneticFilter,Phonetic Filter>>.
+
+Encodes tokens using an improved version of the Soundex algorithm.
+
+See http://en.wikipedia.org/wiki/Soundex
+
+[[PhoneticMatching-Caverphone]]
+== Caverphone
+
+To use this encoding in your analyzer, specify `encoding="Caverphone"` with the <<filter-descriptions.adoc#FilterDescriptions-PhoneticFilter,Phonetic Filter>>.
+
+Caverphone is an algorithm created by the Caversham Project at the University of Otago. The algorithm is optimised for accents present in the southern part of the city of Dunedin, New Zealand.
+
+See http://en.wikipedia.org/wiki/Caverphone and the Caverphone 2.0 specification at http://caversham.otago.ac.nz/files/working/ctp150804.pdf
+
+// OLD_CONFLUENCE_ID: PhoneticMatching-K�lnerPhonetika.k.a.ColognePhonetic
+
+[[PhoneticMatching-K_lnerPhonetika.k.a.ColognePhonetic]]
+== K�lner Phonetik a.k.a. Cologne Phonetic
+
+To use this encoding in your analyzer, specify `encoding="ColognePhonetic"` with the <<filter-descriptions.adoc#FilterDescriptions-PhoneticFilter,Phonetic Filter>>.
+
+The K�lner Phonetik, an algorithm published by Hans Joachim Postel in 1969, is optimized for the German language.
+
+See http://de.wikipedia.org/wiki/K%C3%B6lner_Phonetik
+
+[[PhoneticMatching-NYSIIS]]
+== NYSIIS
+
+To use this encoding in your analyzer, specify `encoding="Nysiis"` with the <<filter-descriptions.adoc#FilterDescriptions-PhoneticFilter,Phonetic Filter>>.
+
+NYSIIS is an encoding used to relate similar names, but can also be used as a general purpose scheme to find words with similar phonemes.
+
+See http://en.wikipedia.org/wiki/NYSIIS and http://www.dropby.com/NYSIIS.html
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/45a148a7/solr/solr-ref-guide/src/ping.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/ping.adoc b/solr/solr-ref-guide/src/ping.adoc
new file mode 100644
index 0000000..942d76d
--- /dev/null
+++ b/solr/solr-ref-guide/src/ping.adoc
@@ -0,0 +1,67 @@
+= Ping
+:page-shortname: ping
+:page-permalink: ping.html
+
+Choosing Ping under a core name issues a `ping` request to check whether the core is up and responding to requests.
+
+image::images/ping/ping.png[image,width=171,height=195]
+
+
+The search executed by a Ping is configured with the <<request-parameters-api.adoc#request-parameters-api,Request Parameters API>>. See <<implicit-requesthandlers.adoc#implicit-requesthandlers,Implicit RequestHandlers>> for the paramset to use for the `/admin/ping` endpoint.
+
+The Ping option doesn't open a page, but the status of the request can be seen on the core overview page shown when clicking on a collection name. The length of time the request has taken is displayed next to the Ping option, in milliseconds.
+
+[[Ping-APIExamples]]
+== API Examples
+
+While the UI screen makes it easy to see the ping response time, the underlying ping command can be more useful when executed by remote monitoring tools:
+
+*Input*
+
+[source,xml]
+----
+http://localhost:8983/solr/<core-name>/admin/ping
+----
+
+This command will ping the core name for a response.
+
+*Input*
+
+[source,xml]
+----
+ http://localhost:8983/solr/<collection-name>admin/ping?wt=json&distrib=true&indent=true
+----
+
+This command will ping all replicas of the given collection name for a response
+
+*Sample Output*
+
+[source,xml]
+----
+<response>
+ <lst name="responseHeader">
+ <int name="status">0</int>
+ <int name="QTime">13</int>
+ <lst name="params">
+ <str name="q">{!lucene}*:*</str>
+ <str name="distrib">false</str>
+ <str name="df">_text_</str>
+ <str name="rows">10</str>
+ <str name="echoParams">all</str>
+ </lst>
+ </lst>
+ <str name="status">OK</str>
+</response>
+----
+
+Both API calls have the same output. A status=OK indicates that the nodes are responding.
+
+*SolrJ Example*
+
+[source,java]
+----
+SolrPing ping = new SolrPing();
+ping.getParams().add("distrib", "true"); //To make it a distributed request against a collection
+rsp = ping.process(solrClient, collectionName);
+int status = rsp.getStatus();
+----
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/45a148a7/solr/solr-ref-guide/src/plugins-stats-screen.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/plugins-stats-screen.adoc b/solr/solr-ref-guide/src/plugins-stats-screen.adoc
new file mode 100644
index 0000000..b76fbec
--- /dev/null
+++ b/solr/solr-ref-guide/src/plugins-stats-screen.adoc
@@ -0,0 +1,14 @@
+= Plugins & Stats Screen
+:page-shortname: plugins-stats-screen
+:page-permalink: plugins-stats-screen.html
+
+The Plugins screen shows information and statistics about the status and performance of various plugins running in each Solr core. You can find information about the performance of the Solr caches, the state of Solr's searchers, and the configuration of Request Handlers and Search Components.
+
+Choose an area of interest on the right, and then drill down into more specifics by clicking on one of the names that appear in the central part of the window. In this example, we've chosen to look at the Searcher stats, from the Core area:
+
+image::images/plugins-stats-screen/plugin-searcher.png[image,width=462,height=250]
+
+
+_Searcher Statistics_
+
+The display is a snapshot taken when the page is loaded. You can get updated status by choosing to either *Watch Changes* or **Refresh Values**. Watching the changes will highlight those areas that have changed, while refreshing the values will reload the page with updated information.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/45a148a7/solr/solr-ref-guide/src/post-tool.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/post-tool.adoc b/solr/solr-ref-guide/src/post-tool.adoc
new file mode 100644
index 0000000..064de5e
--- /dev/null
+++ b/solr/solr-ref-guide/src/post-tool.adoc
@@ -0,0 +1,175 @@
+= Post Tool
+:page-shortname: post-tool
+:page-permalink: post-tool.html
+
+Solr includes a simple command line tool for POSTing various types of content to a Solr server. The tool is `bin/post`. The bin/post tool is a Unix shell script; for Windows (non-Cygwin) usage, see the <<PostTool-WindowsSupport,Windows section>> below.
+
+To run it, open a window and enter:
+
+[source,plain]
+----
+bin/post -c gettingstarted example/films/films.json
+----
+
+This will contact the server at `localhost:8983. S`pecifying the `collection/core name` is **mandatory**. The '-help' (or simply '-h') option will output information on its usage (i.e., `bin/post -help)`.
+
+// OLD_CONFLUENCE_ID: PostTool-Usingthebin/postTool
+
+[[PostTool-Usingthebin_postTool]]
+== Using the bin/post Tool
+
+Specifying either the `collection/core name` or the full update `url` is *mandatory* when using `bin/post`.
+
+The basic usage of `bin/post` is:
+
+[source,plain]
+----
+$ bin/post -h
+Usage: post -c <collection> [OPTIONS] <files|directories|urls|-d ["...",...]>
+ or post -help
+
+ collection name defaults to DEFAULT_SOLR_COLLECTION if not specified
+
+OPTIONS
+=======
+ Solr options:
+ -url <base Solr update URL> (overrides collection, host, and port)
+ -host <host> (default: localhost)
+ -p or -port <port> (default: 8983)
+ -commit yes|no (default: yes)
+ -u or -user <user:pass> (sets BasicAuth credentials)
+
+ Web crawl options:
+ -recursive <depth> (default: 1)
+ -delay <seconds> (default: 10)
+
+
+ Directory crawl options:
+ -delay <seconds> (default: 0)
+
+ stdin/args options:
+ -type <content/type> (default: application/xml)
+
+
+ Other options:
+ -filetypes <type>[,<type>,...] (default: xml,json,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log)
+ -params "<key>=<value>[&<key>=<value>...]" (values must be URL-encoded; these pass through to Solr update request)
+ -out yes|no (default: no; yes outputs Solr response to console)
+...
+----
+
+[[PostTool-Examples]]
+== Examples
+
+There are several ways to use `bin/post`. This section presents several examples.
+
+[[PostTool-IndexingXML]]
+=== Indexing XML
+
+Add all documents with file extension `.xml` to collection or core named `gettingstarted`.
+
+[source,plain]
+----
+ bin/post -c gettingstarted *.xml
+----
+
+Add all documents with file extension `.xml` to the `gettingstarted` collection/core on Solr running on port `8984`.
+
+[source,plain]
+----
+ bin/post -c gettingstarted -p 8984 *.xml
+----
+
+Send XML arguments to delete a document from `gettingstarted`.
+
+[source,plain]
+----
+ bin/post -c gettingstarted -d '<delete><id>42</id></delete>'
+----
+
+[[PostTool-IndexingCSV]]
+=== Indexing CSV
+
+Index all CSV files into `gettingstarted`:
+
+[source,plain]
+----
+ bin/post -c gettingstarted *.csv
+----
+
+Index a tab-separated file into `gettingstarted`:
+
+[source,plain]
+----
+ bin/post -c signals -params "separator=%09" -type text/csv data.tsv
+----
+
+The content type (`-type`) parameter is required to treat the file as the proper type, otherwise it will be ignored and a WARNING logged as it does not know what type of content a .tsv file is. The <<uploading-data-with-index-handlers.adoc#UploadingDatawithIndexHandlers-CSVFormattedIndexUpdates,CSV handler>> supports the `separator` parameter, and is passed through using the `-params` setting.
+
+[[PostTool-IndexingJSON]]
+=== Indexing JSON
+
+Index all JSON files into `gettingstarted`.
+
+[source,plain]
+----
+ bin/post -c gettingstarted *.json
+----
+
+// OLD_CONFLUENCE_ID: PostTool-IndexingRichDocuments(PDF,Word,HTML,etc)
+
+[[PostTool-IndexingRichDocuments_PDF_Word_HTML_etc_]]
+=== Indexing Rich Documents (PDF, Word, HTML, etc)
+
+Index a PDF file into `gettingstarted`.
+
+[source,plain]
+----
+ bin/post -c gettingstarted a.pdf
+----
+
+Automatically detect content types in a folder, and recursively scan it for documents for indexing into `gettingstarted`.
+
+[source,plain]
+----
+ bin/post -c gettingstarted afolder/
+----
+
+Automatically detect content types in a folder, but limit it to PPT and HTML files and index into `gettingstarted`.
+
+[source,plain]
+----
+ bin/post -c gettingstarted -filetypes ppt,html afolder/
+----
+
+// OLD_CONFLUENCE_ID: PostTool-IndexingtoapasswordprotectedSolr(basicauth)
+
+[[PostTool-IndexingtoapasswordprotectedSolr_basicauth_]]
+=== Indexing to a password protected Solr (basic auth)
+
+Index a pdf as the user solr with password `SolrRocks`:
+
+[source,plain]
+----
+ bin/post -u solr:SolrRocks -c gettingstarted a.pdf
+----
+
+[[PostTool-WindowsSupport]]
+== Windows Support
+
+`bin/post` exists currently only as a Unix shell script, however it delegates its work to a cross-platform capable Java program. The <<PostTool-SimplePostTool,`SimplePostTool`>> can be run directly in supported environments, including Windows.
+
+[[PostTool-SimplePostTool]]
+== SimplePostTool
+
+The `bin/post` script currently delegates to a standalone Java program called `SimplePostTool`. This tool, bundled into a executable JAR, can be run directly using `java -jar example/exampledocs/post.jar`. See the help output and take it from there to post files, recurse a website or file system folder, or send direct commands to a Solr server.
+
+[source,plain]
+----
+$ java -jar example/exampledocs/post.jar -h
+SimplePostTool version 5.0.0
+Usage: java [SystemProperties] -jar post.jar [-h|-] [<file|folder|url|arg> [<file|folder|url|arg>...]]
+.
+.
+.
+----