You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by cp...@apache.org on 2017/05/12 13:42:56 UTC
[03/50] [abbrv] lucene-solr:jira/solr-8668: squash merge
jira/solr-10290 into master
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/95968c69/solr/solr-ref-guide/src/v2-api.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/v2-api.adoc b/solr/solr-ref-guide/src/v2-api.adoc
new file mode 100644
index 0000000..12ac623
--- /dev/null
+++ b/solr/solr-ref-guide/src/v2-api.adoc
@@ -0,0 +1,177 @@
+= v2 API
+:page-shortname: v2-api
+:page-permalink: v2-api.html
+
+// TODO: most of the sample introspection calls below include cwiki.apache.org URLs that should be replaced...
+// TODO: ...but they come directly from the API.
+// TODO: so first the core/src/resources/apispec/*.json files need updated, then update the docs to match
+// TODO: What URLs shold the apispec files even point to? version specific or something general?
+
+
+The v2 API is a modernized self-documenting API interface covering most current Solr APIs. It is anticipated that once the v2 API reaches full coverage, and Solr-internal API usages like SolrJ and the Admin UI have been converted from the old API to the v2 API, the old API will eventually be retired.
+
+For now the two API styles will coexist, and all the old APIs will continue to work without any change. You can disable all v2 API endpoints by starting your servers with this system property: `-Ddisable.v2.api=true`.
+
+The old API and the v2 API differ in three principle ways:
+
+1. Command format: The old API commands and associated parameters are provided through URL request parameters on HTTP GET requests, while in the v2 API most API commands are provided via a JSON body POST'ed to v2 API endpoints. The v2 API also supports HTTP methods GET and DELETE where appropriate.
+2. Endpoint structure: The v2 API endpoint structure has been rationalized and regularized.
+3. Documentation: The v2 APIs are self-documenting: append `/_introspect` to any valid v2 API path and the API specification will be returned in JSON format.
+
+[[v2API-v2APIPathPrefixes]]
+== v2 API Path Prefixes
+
+Following are some v2 API URL paths and path prefixes, along with some of the operations that are supported at these paths and their sub-paths.
+
+[width="100%",options="header",]
+|===
+|Path prefix |Some Supported Operations
+|`/v2/collections` or equivalently: `/v2/c` |Create, alias, backup, and restore a collection.
+|`/v2/c/__collection-name__/update` |Update requests.
+|`/v2/c/__collection-name__/config` |Configuration requests.
+|`/v2/c/__collection-name__/schema` |Schema requests.
+|`/v2/c/__collection-name__/__handler-name__` |Handler-specific requests.
+|`/v2/c/__collection-name__/shards` |Split a shard, create a shard, add a replica.
+|`/v2/c/__collection-name__/shards/___shard-name___` |Delete a shard, force leader election
+|`/v2/c/__collection-name__/shards/___shard-name____/____replica-name___` |Delete a replica.
+|`/v2/cores` |Create a core.
+|`/v2/cores/__core-name__` |Reload, rename, delete, and unload a core.
+|`/v2/node` |Perform overseer operation, rejoin leader election.
+|`/v2/cluster` |Add role, remove role, set cluster property.
+|`/v2/c/.system/blob` |Upload and download blobs and metadata.
+|===
+
+[[v2API-Introspect]]
+== Introspect
+
+Append `/_introspect` to any valid v2 API path and the API specification will be returned in JSON format.
+
+`\http://localhost:8983/v2/c/_introspect`
+
+To limit the introspect output to include just one particular HTTP method, add request param `method` with value `GET`, `POST`, or `DELETE`.
+
+`\http://localhost:8983/v2/c/_introspect?method=POST`
+
+Most endpoints support commands provided in a body sent via POST. To limit the introspect output to only one command, add request param `command=__command-name__` .
+
+`\http://localhost:8983/v2/c/gettingstarted/_introspect?method=POST&command=modify`
+
+[[v2API-InterpretingtheIntrospectOutput]]
+=== Interpreting the Introspect Output
+
+Example : `\http://localhost:8983/v2/c/gettingstarted/get/_introspect`
+
+[source,json]
+----
+{
+ "spec":[{
+ "documentation":"https://cwiki.apache.org/confluence/display/solr/RealTime+Get",
+ "description":"RealTime Get allows retrieving documents by ID before the documents have been committed to the index. It is useful when you need access to documents as soon as they are indexed but your commit times are high for other reasons.",
+ "methods":["GET"],
+ "url":{
+ "paths":["/c/gettingstarted/get"],
+ "params":{
+ "id":{
+ "type":"string",
+ "description":"A single document ID to retrieve."},
+ "ids":{
+ "type":"string",
+ "description":"One or more document IDs to retrieve. Separate by commas if more than one ID is specified."},
+ "fq":{
+ "type":"string",
+ "description":"An optional filter query to add to the query. One use case for this is security filtering, in case users or groups should not be able to retrieve the document ID requested."}}}}],
+ "WARNING":"This response format is experimental. It is likely to change in the future.",
+ "availableSubPaths":{}}
+----
+
+Description of some of the keys in the above example:
+
+* `**documentation**` : URL to the online Solr reference guide section for this API
+* `**description**` : A text description of the feature/variable/command etc.
+* `**spec/methods**` : HTTP methods supported by this API
+* `**spec/url/paths**` : URL paths supported by this API
+* `**spec/url/params**` : List of supported URL request params
+* `**availableSubPaths**` : List of valid URL subpaths and the HTTP method(s) each supports
+
+Example of introspect for a POST API: `\http://localhost:8983/v2/c/gettingstarted/_introspect?method=POST&command=modify`
+
+[source,json]
+----
+{
+ "spec":[{
+ "documentation":"https://cwiki.apache.org/confluence/display/solr/Collections+API",
+ "description":"Several collection-level operations are supported with this endpoint: modify collection attributes; reload a collection; migrate documents to a different collection; rebalance collection leaders; balance properties across shards; and add or delete a replica property.",
+ "methods":["POST"],
+ "url":{"paths":["/collections/{collection}",
+ "/c/{collection}"]},
+ "commands":{"modify":{
+ "documentation":"https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-modifycoll",
+ "description":"Modifies specific attributes of a collection. Multiple attributes can be changed at one time.",
+ "type":"object",
+ "properties":{
+ "rule":{
+ "type":"array",
+ "documentation":"https://cwiki.apache.org/confluence/display/solr/Rule-based+Replica+Placement",
+ "description":"Modifies the rules for where replicas should be located in a cluster.",
+ "items":{"type":"string"}},
+ "snitch":{
+ "type":"array",
+ "documentation":"https://cwiki.apache.org/confluence/display/solr/Rule-based+Replica+Placement",
+ "description":"Details of the snitch provider",
+ "items":{"type":"string"}},
+ "autoAddReplicas":{
+ "type":"boolean",
+ "description":"When set to true, enables auto addition of replicas on shared file systems (such as HDFS). See https://cwiki.apache.org/confluence/display/solr/Running+Solr+on+HDFS for more details on settings and overrides."},
+ "replicationFactor":{
+ "type":"string",
+ "description":"The number of replicas to be created for each shard. Replicas are physical copies of each shard, acting as failover for the shard. Note that changing this value on an existing collection does not automatically add more replicas to the collection. However, it will allow add-replica commands to succeed."},
+ "maxShardsPerNode":{
+ "type":"integer",
+ "description":"When creating collections, the shards and/or replicas are spread across all available, live, nodes, and two replicas of the same shard will never be on the same node. If a node is not live when the collection is created, it will not get any parts of the new collection, which could lead to too many replicas being created on a single live node. Defining maxShardsPerNode sets a limit on the number of replicas can be spread to each node. If the entire collection can not be fit into the live nodes, no collection will be created at all."}}}}}],
+ "WARNING":"This response format is experimental. It is likely to change in the future.",
+ "availableSubPaths":{
+ "/c/gettingstarted/select":["POST", "GET"],
+ "/c/gettingstarted/config":["POST", "GET"],
+ "/c/gettingstarted/schema":["POST", "GET"],
+ "/c/gettingstarted/export":["POST", "GET"],
+ "/c/gettingstarted/admin/ping":["POST", "GET"],
+ "/c/gettingstarted/update":["POST"]},
+
+[... more sub-paths ...]
+
+}
+----
+
+The `"commands"` section in the above example has one entry for each command supported at this endpoint. The key is the command name and the value is a json object describing the command structure using JSON schema (see http://json-schema.org/ for a description).
+
+[[v2API-InvocationExamples]]
+== Invocation Examples
+
+For the "gettingstarted" collection, set the replication factor and whether to automatically add replicas (see above for the introspect output for the `"modify"` command used here):
+
+[source,bash]
+----
+$ curl http://localhost:8983/v2/c/gettingstarted -H 'Content-type:application/json' -d '
+{ modify: { replicationFactor: "3", autoAddReplicas: false } }'
+
+{"responseHeader":{"status":0,"QTime":842}}
+----
+
+See the state of the cluster:
+
+[source,bash]
+----
+$ curl http://localhost:8983/v2/cluster
+
+{"responseHeader":{"status":0,"QTime":0},"collections":["gettingstarted",".system"]}
+----
+
+Set a cluster property:
+
+[source,bash]
+----
+$ curl http://localhost:8983/v2/cluster -H 'Content-type: application/json' -d '
+{ set-property: { name: autoAddReplicas, val: "false" } }'
+
+{"responseHeader":{"status":0,"QTime":4}}
+----
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/95968c69/solr/solr-ref-guide/src/velocity-response-writer.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/velocity-response-writer.adoc b/solr/solr-ref-guide/src/velocity-response-writer.adoc
new file mode 100644
index 0000000..3f3b32c
--- /dev/null
+++ b/solr/solr-ref-guide/src/velocity-response-writer.adoc
@@ -0,0 +1,107 @@
+= Velocity Response Writer
+:page-shortname: velocity-response-writer
+:page-permalink: velocity-response-writer.html
+
+The VelocityResponseWriter is an optional plugin available in the `contrib/velocity` directory. It powers the /browse user interfaces when using configurations such as "basic_configs", "techproducts", and "example/files".
+
+Its JAR and dependencies must be added (via `<lib>` or solr/home lib inclusion), and must be registered in `solrconfig.xml` like this:
+
+[source,xml]
+----
+<queryResponseWriter name="velocity" class="solr.VelocityResponseWriter">
+ <str name="template.base.dir">${velocity.template.base.dir:}</str>
+
+<!--
+ <str name="init.properties.file">velocity-init.properties</str>
+ <bool name="params.resource.loader.enabled">true</bool>
+ <bool name="solr.resource.loader.enabled">false</bool>
+ <lst name="tools">
+ <str name="mytool">com.example.MyCustomTool</str>
+ </lst>
+-->
+</queryResponseWriter>
+----
+
+The above example shows the optional initialization and custom tool parameters used by VelocityResponseWriter; these are detailed in the following table. These initialization parameters are only specified in the writer registration in solrconfig.xml, not as request-time parameters. See further below for request-time parameters.
+
+== Configuration & Usage
+
+[[VelocityResponseWriter-VelocityResponseWriterinitializationparameters]]
+=== VelocityResponseWriter Initialization Parameters
+
+// TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
+
+[cols="20,60,20",options="header"]
+|===
+|Parameter |Description |Default value
+|template.base.dir |If specified and exists as a file system directory, a file resource loader will be added for this directory. Templates in this directory will override "solr" resource loader templates. |
+|init.properties.file |Specifies a properties file name which must exist in the Solr `conf/` directory (**not** under a `velocity/` subdirectory) or root of a JAR file in a <lib>. |
+|params.resource.loader.enabled a|
+The "params" resource loader allows templates to be specified in Solr request parameters. For example:
+
+`\http://localhost:8983/solr/gettingstarted/select?q=\*:*&wt=velocity&v.template=custom&v.template.custom=CUSTOM%3A%20%23core_name`
+
+where `v.template=custom` says to render a template called "custom" and `v.template.custom`'s value is the actual custom template. This is disabled by default; it'd be a niche, unusual, use case to need this enabled.
+
+ |false
+|solr.resource.loader.enabled |The "solr" resource loader is the only template loader registered by default. Templates are served from resources visible to the SolrResourceLoader under a `velocity/` subdirectory. The VelocityResponseWriter itself has some built-in templates (in its JAR file, under velocity/) that are available automatically through this loader. These built-in templates can be overridden when the same template name is in conf/velocity/ or by using the `template.base.dir` option. |true
+|tools |External "tools" can be specified as list of string name/value (tool name / class name) pairs. Tools, in the Velocity context, are simply Java objects. Tool classes are constructed using a no-arg constructor (or a single-SolrCore-arg constructor if it exists) and added to the Velocity context with the specified name. A custom registered tool can override the built-in context objects with the same name, except for $request, $response, $page, and $debug (these tools are designed to not be overridden). |
+|===
+
+[[VelocityResponseWriter-VelocityResponseWriterrequestparameters]]
+=== VelocityResponseWriter Request Parameters
+
+// TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
+
+[cols="20,60,20",options="header"]
+|===
+|Parameter |Description |Default value
+|v.template |Specifies the name of the template to render. |
+|v.layout a|
+Specifies a template name to use as the layout around the main, `v.template`, specified template.
+
+The main template is rendered into a string value included into the layout rendering as `$content`.
+
+ |
+|v.layout.enabled |Determines if the main template should have a layout wrapped around it. True by default, but requires `v.layout` to specified as well. |true
+|v.contentType |Specifies the content type used in the HTTP response. If not specified, the default will depend on whether `v.json` is specified or not. a|
+without json.wrf: text/html;charset=UTF-8
+
+with json.wrf: application/json;charset=UTF-8
+
+|v.json a|
+Specifies a function name to wrap around the response rendered as JSON. If specified, the content type used in the response will be "application/json;charset=UTF-8", unless overridden by `v.contentType`.
+
+Output will be in this format (with v.json=wrf):
+
+`wrf("result":"<Velocity generated response string, with quotes and backslashes escaped>")`
+
+ |
+|v.locale |Locale to use with the `$resource` tool and other LocaleConfig implementing tools. The default locale is `Locale.ROOT`. Localized resources are loaded from standard Java resource bundles named `resources[_locale-code].properties`. Resource bundles can be added by providing a JAR file visible by the SolrResourceLoader with resource bundles under a velocity sub-directory. Resource bundles are not loadable under conf/, as only the class loader aspect of SolrResourceLoader can be used here. |
+|v.template.<template_name> |When the "params" resource loader is enabled, templates can be specified as part of the Solr request. |
+|===
+
+[[VelocityResponseWriter-VelocityResponseWritercontextobjects]]
+=== VelocityResponseWriter Context Objects
+
+// TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
+
+[cols="30,70",options="header"]
+|===
+|Context Reference |Description
+|request |http://lucene.apache.org/solr/api/org/apache/solr/request/SolrQueryRequest.html[SolrQueryRequest] javadocs
+|response |http://lucene.apache.org/solr/api/org/apache/solr/client/solrj/response/QueryResponse.html[QueryResponse] most of the time, but in some cases where https://wiki.apache.org/solr/QueryResponse[QueryResponse] doesn't like the request handlers output (https://wiki.apache.org/solr/AnalysisRequestHandler[AnalysisRequestHandler], for example, causes a ClassCastException parsing "response"), the response will be a https://wiki.apache.org/solr/SolrResponseBase[SolrResponseBase] object.
+|esc |A Velocity http://velocity.apache.org/tools/2.0/tools-summary.html#EscapeTool[EscapeTool] instance
+|date |A Velocity http://velocity.apache.org/tools/2.0/tools-summary.html#ComparisonDateTool[ComparisonDateTool] instance
+|list |A Velocity http://velocity.apache.org/tools/2.0/apidocs/org/apache/velocity/tools/generic/ListTool.html[ListTool] instance
+|math |A Velocity http://velocity.apache.org/tools/2.0/tools-summary.html#MathTool[MathTool] instance
+|number |A Velocity http://velocity.apache.org/tools/2.0/tools-summary.html#NumberTool[NumberTool] instance
+|sort |A Velocity http://velocity.apache.org/tools/2.0/tools-summary.html#SortTool[SortTool] instance
+|display |A Velocity http://velocity.apache.org/tools/2.0/tools-summary.html#DisplayTool[DisplayTool] instance
+|resource |A Velocity http://velocity.apache.org/tools/2.0/tools-summary.html#ResourceTool[ResourceTool] instance
+|engine |The current VelocityEngine instance
+|page |An instance of Solr's PageTool (only included if the response is a QueryResponse where paging makes sense)
+|debug |A shortcut to the debug part of the response, or null if debug is not on. This is handy for having debug-only sections in a template using `#if($debug)...#end`
+|content |The rendered output of the main template, when rendering the layout (v.layout.enabled=true and v.layout=<template>).
+|[custom tool(s)] |Tools provided by the optional "tools" list of the VelocityResponseWriter registration are available by their specified name.
+|===
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/95968c69/solr/solr-ref-guide/src/velocity-search-ui.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/velocity-search-ui.adoc b/solr/solr-ref-guide/src/velocity-search-ui.adoc
new file mode 100644
index 0000000..f41a204
--- /dev/null
+++ b/solr/solr-ref-guide/src/velocity-search-ui.adoc
@@ -0,0 +1,12 @@
+= Velocity Search UI
+:page-shortname: velocity-search-ui
+:page-permalink: velocity-search-ui.html
+
+Solr includes a sample search UI based on the <<response-writers.adoc#ResponseWriters-VelocityResponseWriter,VelocityResponseWriter>> (also known as Solritas) that demonstrates several useful features, such as searching, faceting, highlighting, autocomplete, and geospatial searching.
+
+When using the `sample_techproducts_configs` config set, you can access the Velocity sample Search UI: `\http://localhost:8983/solr/techproducts/browse`
+
+.The Velocity Search UI
+image::images/velocity-search-ui/techproducts_browse.png[image,width=500]
+
+For more information about the Velocity Response Writer, see the <<response-writers.adoc#ResponseWriters-VelocityResponseWriter,Response Writer page>>.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/95968c69/solr/solr-ref-guide/src/working-with-currencies-and-exchange-rates.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/working-with-currencies-and-exchange-rates.adoc b/solr/solr-ref-guide/src/working-with-currencies-and-exchange-rates.adoc
new file mode 100644
index 0000000..b5d8e0c
--- /dev/null
+++ b/solr/solr-ref-guide/src/working-with-currencies-and-exchange-rates.adoc
@@ -0,0 +1,88 @@
+= Working with Currencies and Exchange Rates
+:page-shortname: working-with-currencies-and-exchange-rates
+:page-permalink: working-with-currencies-and-exchange-rates.html
+
+The `currency` FieldType provides support for monetary values to Solr/Lucene with query-time currency conversion and exchange rates. The following features are supported:
+
+* Point queries
+* Range queries
+* Function range queries
+* Sorting
+* Currency parsing by either currency code or symbol
+* Symmetric & asymmetric exchange rates (asymmetric exchange rates are useful if there are fees associated with exchanging the currency)
+
+[[WorkingwithCurrenciesandExchangeRates-ConfiguringCurrencies]]
+== Configuring Currencies
+
+The `currency` field type is defined in `schema.xml`. This is the default configuration of this type:
+
+[source,xml]
+----
+<fieldType name="currency" class="solr.CurrencyField" precisionStep="8"
+ defaultCurrency="USD" currencyConfig="currency.xml" />
+----
+
+In this example, we have defined the name and class of the field type, and defined the `defaultCurrency` as "USD", for U.S. Dollars. We have also defined a `currencyConfig` to use a file called "currency.xml". This is a file of exchange rates between our default currency to other currencies. There is an alternate implementation that would allow regular downloading of currency data. See <<WorkingwithCurrenciesandExchangeRates-ExchangeRates,Exchange Rates>> below for more.
+
+Many of the example schemas that ship with Solr include a <<dynamic-fields.adoc#dynamic-fields,dynamic field>> that uses this type, such as this example:
+
+[source,xml]
+----
+ <dynamicField name="*_c" type="currency" indexed="true" stored="true"/>
+----
+
+This dynamic field would match any field that ends in `_c` and make it a currency typed field.
+
+At indexing time, money fields can be indexed in a native currency. For example, if a product on an e-commerce site is listed in Euros, indexing the price field as "1000,EUR" will index it appropriately. The price should be separated from the currency by a comma, and the price must be encoded with a floating point value (a decimal point).
+
+During query processing, range and point queries are both supported.
+
+[[WorkingwithCurrenciesandExchangeRates-ExchangeRates]]
+== Exchange Rates
+
+You configure exchange rates by specifying a provider. Natively, two provider types are supported: `FileExchangeRateProvider` or `OpenExchangeRatesOrgProvider`.
+
+[[WorkingwithCurrenciesandExchangeRates-FileExchangeRateProvider]]
+=== FileExchangeRateProvider
+
+This provider requires you to provide a file of exchange rates. It is the default, meaning that to use this provider you only need to specify the file path and name as a value for `currencyConfig` in the definition for this type.
+
+There is a sample `currency.xml` file included with Solr, found in the same directory as the `schema.xml` file. Here is a small snippet from this file:
+
+[source,xml]
+----
+<currencyConfig version="1.0">
+ <rates>
+ <!-- Updated from http://www.exchangerate.com/ at 2011-09-27 -->
+ <rate from="USD" to="ARS" rate="4.333871" comment="ARGENTINA Peso" />
+ <rate from="USD" to="AUD" rate="1.025768" comment="AUSTRALIA Dollar" />
+ <rate from="USD" to="EUR" rate="0.743676" comment="European Euro" />
+ <rate from="USD" to="CAD" rate="1.030815" comment="CANADA Dollar" />
+
+ <!-- Cross-rates for some common currencies -->
+ <rate from="EUR" to="GBP" rate="0.869914" />
+ <rate from="EUR" to="NOK" rate="7.800095" />
+ <rate from="GBP" to="NOK" rate="8.966508" />
+
+ <!-- Asymmetrical rates -->
+ <rate from="EUR" to="USD" rate="0.5" />
+ </rates>
+</currencyConfig>
+----
+
+[[WorkingwithCurrenciesandExchangeRates-OpenExchangeRatesOrgProvider]]
+=== OpenExchangeRatesOrgProvider
+
+You can configure Solr to download exchange rates from http://www.OpenExchangeRates.Org[OpenExchangeRates.Org], with updates rates between USD and 170 currencies hourly. These rates are symmetrical only.
+
+In this case, you need to specify the `providerClass` in the definitions for the field type and sign up for an API key. Here is an example:
+
+[source,xml]
+----
+<fieldType name="currency" class="solr.CurrencyField" precisionStep="8"
+ providerClass="solr.OpenExchangeRatesOrgProvider"
+ refreshInterval="60"
+ ratesFileLocation="http://www.openexchangerates.org/api/latest.json?app_id=yourPersonalAppIdKey"/>
+----
+
+The `refreshInterval` is minutes, so the above example will download the newest rates every 60 minutes. The refresh interval may be increased, but not decreased.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/95968c69/solr/solr-ref-guide/src/working-with-dates.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/working-with-dates.adoc b/solr/solr-ref-guide/src/working-with-dates.adoc
new file mode 100644
index 0000000..22959b7
--- /dev/null
+++ b/solr/solr-ref-guide/src/working-with-dates.adoc
@@ -0,0 +1,157 @@
+= Working with Dates
+:page-shortname: working-with-dates
+:page-permalink: working-with-dates.html
+
+[[WorkingwithDates-DateFormatting]]
+== Date Formatting
+
+Solr's date fields (`TrieDateField`, `DatePointField` and `DateRangeField`) represent "dates" as a point in time with millisecond precision. The format used is a restricted form of the canonical representation of dateTime in the http://www.w3.org/TR/xmlschema-2/#dateTime[XML Schema specification] – a restricted subset of https://en.wikipedia.org/wiki/ISO_8601[ISO-8601]. For those familiar with Java 8, Solr uses https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#ISO_INSTANT[DateTimeFormatter.ISO_INSTANT] for formatting, and parsing too with "leniency".
+
+`YYYY-MM-DDThh:mm:ssZ`
+
+* `YYYY` is the year.
+* `MM` is the month.
+* `DD` is the day of the month.
+* `hh` is the hour of the day as on a 24-hour clock.
+* `mm` is minutes.
+* `ss` is seconds.
+* `Z` is a literal 'Z' character indicating that this string representation of the date is in UTC
+
+Note that no time zone can be specified; the String representations of dates is always expressed in Coordinated Universal Time (UTC). Here is an example value:
+
+`1972-05-20T17:33:18Z`
+
+You can optionally include fractional seconds if you wish, although any precision beyond milliseconds will be ignored. Here are example values with sub-seconds:
+
+* `1972-05-20T17:33:18.772Z`
+* `1972-05-20T17:33:18.77Z`
+* `1972-05-20T17:33:18.7Z`
+
+There must be a leading '`-`' for dates prior to year 0000, and Solr will format dates with a leading '`+`' for years after 9999. Year 0000 is considered year 1 BC; there is no such thing as year 0 AD or BC.
+
+.Query escaping may be required
+[WARNING]
+====
+
+As you can see, the date format includes colon characters separating the hours, minutes, and seconds. Because the colon is a special character to Solr's most common query parsers, escaping is sometimes required, depending on exactly what you are trying to do.
+
+This is normally an invalid query: `datefield:1972-05-20T17:33:18.772Z`
+
+These are valid queries: `datefield:1972-05-20T17\:33\:18.772Z` `datefield:"1972-05-20T17:33:18.772Z"` `datefield:[1972-05-20T17:33:18.772Z TO *]`
+
+====
+
+[[WorkingwithDates-DateRangeFormatting]]
+=== Date Range Formatting
+
+Solr's `DateRangeField` supports the same point in time date syntax described above (with _date math_ described below) and more to express date ranges. One class of examples is truncated dates, which represent the entire date span to the precision indicated. The other class uses the range syntax (`[ TO ]`). Here are some examples:
+
+* `2000-11` – The entire month of November, 2000.
+* `2000-11T13` – Likewise but for an hour of the day (1300 to before 1400, i.e. 1pm to 2pm).
+* `-0009` – The year 10 BC. A 0 in the year position is 0 AD, and is also considered 1 BC.
+* `[2000-11-01 TO 2014-12-01]` – The specified date range at a day resolution.
+* `[2014 TO 2014-12-01]` – From the start of 2014 till the end of the first day of December.
+* `[* TO 2014-12-01]` – From the earliest representable time thru till the end of the day on 2014-12-01.
+
+Limitations: The range syntax doesn't support embedded date math. If you specify a date instance supported by TrieDateField with date math truncating it, like `NOW/DAY`, you still get the first millisecond of that day, not the entire day's range. Exclusive ranges (using `{` & `}`) work in _queries_ but not for _indexing_ ranges.
+
+[[WorkingwithDates-DateMath]]
+== Date Math
+
+Solr's date field types also supports _date math_ expressions, which makes it easy to create times relative to fixed moments in time, include the current time which can be represented using the special value of "```NOW```".
+
+[[WorkingwithDates-DateMathSyntax]]
+=== Date Math Syntax
+
+Date math expressions consist either adding some quantity of time in a specified unit, or rounding the current time by a specified unit. expressions can be chained and are evaluated left to right.
+
+For example: this represents a point in time two months from now:
+
+`NOW+2MONTHS`
+
+This is one day ago:
+
+`NOW-1DAY`
+
+A slash is used to indicate rounding. This represents the beginning of the current hour:
+
+`NOW/HOUR`
+
+The following example computes (with millisecond precision) the point in time six months and three days into the future and then rounds that time to the beginning of that day:
+
+`NOW+6MONTHS+3DAYS/DAY`
+
+Note that while date math is most commonly used relative to `NOW` it can be applied to any fixed moment in time as well:
+
+`1972-05-20T17:33:18.772Z+6MONTHS+3DAYS/DAY`
+
+[[WorkingwithDates-RequestParametersThatAffectDateMath]]
+=== Request Parameters That Affect Date Math
+
+[[WorkingwithDates-NOW]]
+==== `NOW`
+
+The `NOW` parameter is used internally by Solr to ensure consistent date math expression parsing across multiple nodes in a distributed request. But it can be specified to instruct Solr to use an arbitrary moment in time (past or future) to override for all situations where the the special value of "```NOW```" would impact date math expressions.
+
+It must be specified as a (long valued) milliseconds since epoch
+
+Example:
+
+`q=solr&fq=start_date:[* TO NOW]&NOW=1384387200000`
+
+[[WorkingwithDates-TZ]]
+==== `TZ`
+
+By default, all date math expressions are evaluated relative to the UTC TimeZone, but the `TZ` parameter can be specified to override this behaviour, by forcing all date based addition and rounding to be relative to the specified http://docs.oracle.com/javase/8/docs/api/java/util/TimeZone.html[time zone].
+
+For example, the following request will use range faceting to facet over the current month, "per day" relative UTC:
+
+[source,text]
+----
+http://localhost:8983/solr/my_collection/select?q=*:*&facet.range=my_date_field&facet=true&facet.range.start=NOW/MONTH&facet.range.end=NOW/MONTH%2B1MONTH&facet.range.gap=%2B1DAY
+----
+
+[source,xml]
+----
+<int name="2013-11-01T00:00:00Z">0</int>
+<int name="2013-11-02T00:00:00Z">0</int>
+<int name="2013-11-03T00:00:00Z">0</int>
+<int name="2013-11-04T00:00:00Z">0</int>
+<int name="2013-11-05T00:00:00Z">0</int>
+<int name="2013-11-06T00:00:00Z">0</int>
+<int name="2013-11-07T00:00:00Z">0</int>
+...
+----
+
+While in this example, the "days" will be computed relative to the specified time zone - including any applicable Daylight Savings Time adjustments:
+
+[source,text]
+----
+http://localhost:8983/solr/my_collection/select?q=*:*&facet.range=my_date_field&facet=true&facet.range.start=NOW/MONTH&facet.range.end=NOW/MONTH%2B1MONTH&facet.range.gap=%2B1DAY&TZ=America/Los_Angeles
+----
+
+[source,xml]
+----
+<int name="2013-11-01T07:00:00Z">0</int>
+<int name="2013-11-02T07:00:00Z">0</int>
+<int name="2013-11-03T07:00:00Z">0</int>
+<int name="2013-11-04T08:00:00Z">0</int>
+<int name="2013-11-05T08:00:00Z">0</int>
+<int name="2013-11-06T08:00:00Z">0</int>
+<int name="2013-11-07T08:00:00Z">0</int>
+...
+----
+
+[[WorkingwithDates-MoreDateRangeFieldDetails]]
+== More DateRangeField Details
+
+`DateRangeField` is almost a drop-in replacement for places where `TrieDateField` is used. The only difference is that Solr's XML or SolrJ response formats will expose the stored data as a String instead of a Date. The underlying index data for this field will be a bit larger. Queries that align to units of time a second on up should be faster than TrieDateField, especially if it's in UTC. But the main point of DateRangeField as its name suggests is to allow indexing date ranges. To do that, simply supply strings in the format shown above. It also supports specifying 3 different relational predicates between the indexed data, and the query range: `Intersects` (default), `Contains`, `Within`. You can specify the predicate by querying using the `op` local-params parameter like so:
+
+[source,text]
+----
+fq={!field f=dateRange op=Contains}[2013 TO 2018]
+----
+
+Unlike most/all local-params, `op` is actually _not_ defined by any query parser (`field`), it is defined by the field type – `DateRangeField`. In that example, it would find documents with indexed ranges that _contain_ (or equals) the range 2013 thru 2018. Multi-valued overlapping indexed ranges in a document are effectively coalesced.
+
+For a DateRangeField example use-case and possibly other information, http://wiki.apache.org/solr/DateRangeField[see Solr's community wiki].
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/95968c69/solr/solr-ref-guide/src/working-with-enum-fields.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/working-with-enum-fields.adoc b/solr/solr-ref-guide/src/working-with-enum-fields.adoc
new file mode 100644
index 0000000..0c7963b
--- /dev/null
+++ b/solr/solr-ref-guide/src/working-with-enum-fields.adoc
@@ -0,0 +1,60 @@
+= Working with Enum Fields
+:page-shortname: working-with-enum-fields
+:page-permalink: working-with-enum-fields.html
+
+The EnumField type allows defining a field whose values are a closed set, and the sort order is pre-determined but is not alphabetic nor numeric. Examples of this are severity lists, or risk definitions.
+
+[[WorkingwithEnumFields-DefininganEnumFieldinschema.xml]]
+== Defining an EnumField in `schema.xml`
+
+The EnumField type definition is quite simple, as in this example defining field types for "priorityLevel" and "riskLevel" enumerations:
+
+[source,xml]
+----
+<fieldType name="priorityLevel" class="solr.EnumField" enumsConfig="enumsConfig.xml" enumName="priority"/>
+<fieldType name="riskLevel" class="solr.EnumField" enumsConfig="enumsConfig.xml" enumName="risk" />
+----
+
+Besides the `name` and the `class`, which are common to all field types, this type also takes two additional parameters:
+
+* `enumsConfig`: the name of a configuration file that contains the `<enum/>` list of field values and their order that you wish to use with this field type. If a path to the file is not defined specified, the file should be in the `conf` directory for the collection.
+* `enumName`: the name of the specific enumeration in the `enumsConfig` file to use for this type.
+
+[[WorkingwithEnumFields-DefiningtheEnumFieldconfigurationfile]]
+== Defining the EnumField configuration file
+
+The file named with the `enumsConfig` parameter can contain multiple enumeration value lists with different names if there are multiple uses for enumerations in your Solr schema.
+
+In this example, there are two value lists defined. Each list is between `enum` opening and closing tags:
+
+[source,xml]
+----
+<?xml version="1.0" ?>
+<enumsConfig>
+ <enum name="priority">
+ <value>Not Available</value>
+ <value>Low</value>
+ <value>Medium</value>
+ <value>High</value>
+ <value>Urgent</value>
+ </enum>
+ <enum name="risk">
+ <value>Unknown</value>
+ <value>Very Low</value>
+ <value>Low</value>
+ <value>Medium</value>
+ <value>High</value>
+ <value>Critical</value>
+ </enum>
+</enumsConfig>
+----
+
+.Changing Values
+[IMPORTANT]
+====
+
+You cannot change the order, or remove, existing values in an `<enum/>` without reindexing.
+
+You can however add new values to the end.
+
+====
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/95968c69/solr/solr-ref-guide/src/working-with-external-files-and-processes.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/working-with-external-files-and-processes.adoc b/solr/solr-ref-guide/src/working-with-external-files-and-processes.adoc
new file mode 100644
index 0000000..cc4665c
--- /dev/null
+++ b/solr/solr-ref-guide/src/working-with-external-files-and-processes.adoc
@@ -0,0 +1,294 @@
+= Working with External Files and Processes
+:page-shortname: working-with-external-files-and-processes
+:page-permalink: working-with-external-files-and-processes.html
+
+[[WorkingwithExternalFilesandProcesses-TheExternalFileFieldType]]
+== The `ExternalFileField` Type
+
+The `ExternalFileField` type makes it possible to specify the values for a field in a file outside the Solr index. For such a field, the file contains mappings from a key field to the field value. Another way to think of this is that, instead of specifying the field in documents as they are indexed, Solr finds values for this field in the external file.
+
+[IMPORTANT]
+====
+External fields are not searchable. They can be used only for function queries or display. For more information on function queries, see the section on <<function-queries.adoc#function-queries,Function Queries>>.
+====
+
+The `ExternalFileField` type is handy for cases where you want to update a particular field in many documents more often than you want to update the rest of the documents. For example, suppose you have implemented a document rank based on the number of views. You might want to update the rank of all the documents daily or hourly, while the rest of the contents of the documents might be updated much less frequently. Without `ExternalFileField`, you would need to update each document just to change the rank. Using `ExternalFileField` is much more efficient because all document values for a particular field are stored in an external file that can be updated as frequently as you wish.
+
+In `schema.xml`, the definition of this field type might look like this:
+
+[source,xml]
+----
+<fieldType name="entryRankFile" keyField="pkId" defVal="0" stored="false" indexed="false" class="solr.ExternalFileField" valType="pfloat"/>
+----
+
+The `keyField` attribute defines the key that will be defined in the external file. It is usually the unique key for the index, but it doesn't need to be as long as the `keyField` can be used to identify documents in the index. A `defVal` defines a default value that will be used if there is no entry in the external file for a particular document.
+
+The `valType` attribute specifies the actual type of values that will be found in the file. The type specified must be either a float field type, so valid values for this attribute are `pfloat`, `float` or `tfloat`. This attribute can be omitted.
+
+[[WorkingwithExternalFilesandProcesses-FormatoftheExternalFile]]
+=== Format of the External File
+
+The file itself is located in Solr's index directory, which by default is `$SOLR_HOME/data`. The name of the file should be `external___fieldname__` or `external___fieldname__.*`. For the example above, then, the file could be named `external_entryRankFile` or `external_entryRankFile.txt`.
+
+[TIP]
+====
+If any files using the name pattern `.*` (such as `.txt`) appear, the last (after being sorted by name) will be used and previous versions will be deleted. This behavior supports implementations on systems where one may not be able to overwrite a file (for example, on Windows, if the file is in use).
+====
+
+The file contains entries that map a key field, on the left of the equals sign, to a value, on the right. Here are a few example entries:
+
+[source,text]
+----
+doc33=1.414
+doc34=3.14159
+doc40=42
+----
+
+The keys listed in this file do not need to be unique. The file does not need to be sorted, but Solr will be able to perform the lookup faster if it is.
+
+[[WorkingwithExternalFilesandProcesses-ReloadinganExternalFile]]
+=== Reloading an External File
+
+It's possible to define an event listener to reload an external file when either a searcher is reloaded or when a new searcher is started. See the section <<query-settings-in-solrconfig.adoc#QuerySettingsinSolrConfig-Query-RelatedListeners,Query-Related Listeners>> for more information, but a sample definition in `solrconfig.xml` might look like this:
+
+[source,xml]
+----
+<listener event="newSearcher" class="org.apache.solr.schema.ExternalFileFieldReloader"/>
+<listener event="firstSearcher" class="org.apache.solr.schema.ExternalFileFieldReloader"/>
+----
+
+[[WorkingwithExternalFilesandProcesses-ThePreAnalyzedFieldType]]
+== The `PreAnalyzedField` Type
+
+The `PreAnalyzedField` type provides a way to send to Solr serialized token streams, optionally with independent stored values of a field, and have this information stored and indexed without any additional text processing applied in Solr. This is useful if user wants to submit field content that was already processed by some existing external text processing pipeline (e.g., it has been tokenized, annotated, stemmed, synonyms inserted, etc.), while using all the rich attributes that Lucene's TokenStream provides (per-token attributes).
+
+The serialization format is pluggable using implementations of PreAnalyzedParser interface. There are two out-of-the-box implementations:
+
+* <<WorkingwithExternalFilesandProcesses-JsonPreAnalyzedParser,JsonPreAnalyzedParser>>: as the name suggests, it parses content that uses JSON to represent field's content. This is the default parser to use if the field type is not configured otherwise.
+* <<WorkingwithExternalFilesandProcesses-SimplePreAnalyzedParser,SimplePreAnalyzedParser>>: uses a simple strict plain text format, which in some situations may be easier to create than JSON.
+
+There is only one configuration parameter, `parserImpl`. The value of this parameter should be a fully qualified class name of a class that implements PreAnalyzedParser interface. The default value of this parameter is `org.apache.solr.schema.JsonPreAnalyzedParser`.
+
+By default, the query-time analyzer for fields of this type will be the same as the index-time analyzer, which expects serialized pre-analyzed text. You must add a query type analyzer to your fieldType in order to perform analysis on non-pre-analyzed queries. In the example below, the index-time analyzer expects the default JSON serialization format, and the query-time analyzer will employ StandardTokenizer/LowerCaseFilter:
+
+[source,xml]
+----
+<fieldType name="pre_with_query_analyzer" class="solr.PreAnalyzedField">
+ <analyzer type="query">
+ <tokenizer class="solr.StandardTokenizerFactory"/>
+ <filter class="solr.LowerCaseFilterFactory"/>
+ </analyzer>
+</fieldType>
+----
+
+[[WorkingwithExternalFilesandProcesses-JsonPreAnalyzedParser]]
+=== JsonPreAnalyzedParser
+
+This is the default serialization format used by PreAnalyzedField type. It uses a top-level JSON map with the following keys:
+
+// TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
+
+[cols="20,60,20",options="header"]
+|===
+|Key |Description |Required
+|`v` |Version key. Currently the supported version is `1`. |required
+|`str` |Stored string value of a field. You can use at most one of `str` or `bin`. |optional
+|`bin` |Stored binary value of a field. The binary value has to be Base64 encoded. |optional
+|`tokens` |serialized token stream. This is a JSON list. |optional
+|===
+
+Any other top-level key is silently ignored.
+
+[[WorkingwithExternalFilesandProcesses-Tokenstreamserialization]]
+==== Token stream serialization
+
+The token stream is expressed as a JSON list of JSON maps. The map for each token consists of the following keys and values:
+
+// TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
+
+[cols="10,20,20,30,20",options="header"]
+|===
+|Key |Description |Lucene Attribute |Value |Required?
+|`t` |token |{lucene-javadocs}/core/org/apache/lucene/analysis/tokenattributes/CharTermAttribute.html[CharTermAttribute] |UTF-8 string representing the current token |required
+|`s` |start offset |{lucene-javadocs}/core/org/apache/lucene/analysis/tokenattributes/OffsetAttribute.html[OffsetAttribute] |Non-negative integer |optional
+|`e` |end offset |OffsetAttribute |Non-negative integer |optional
+|`i` |position increment |{lucene-javadocs}/core/org/apache/lucene/analysis/tokenattributes/PositionIncrementAttribute.html[PositionIncrementAttribute] |Non-negative integer - default is `1` |optional
+|`p` |payload |{lucene-javadocs}/core/org/apache/lucene/analysis/tokenattributes/PayloadAttribute.html[PayloadAttribute] |Base64 encoded payload |optional
+|`y` |lexical type |{lucene-javadocs}/core/org/apache/lucene/analysis/tokenattributes/TypeAttribute.html[TypeAttribute] |UTF-8 string |optional
+|`f` |flags |{lucene-javadocs}/core/org/apache/lucene/analysis/tokenattributes/FlagsAttribute.html[FlagsAttribute] |String representing an integer value in hexadecimal format |optional
+|===
+
+Any other key is silently ignored.
+
+[[WorkingwithExternalFilesandProcesses-Example]]
+==== Example
+
+[source,json]
+----
+{
+ "v":"1",
+ "str":"test ąćęłńóśźż",
+ "tokens": [
+ {"t":"one","s":123,"e":128,"i":22,"p":"DQ4KDQsODg8=","y":"word"},
+ {"t":"two","s":5,"e":8,"i":1,"y":"word"},
+ {"t":"three","s":20,"e":22,"i":1,"y":"foobar"}
+ ]
+}
+----
+
+[[WorkingwithExternalFilesandProcesses-SimplePreAnalyzedParser]]
+=== SimplePreAnalyzedParser
+
+The fully qualified class name to use when specifying this format via the `parserImpl` configuration parameter is `org.apache.solr.schema.SimplePreAnalyzedParser`.
+
+[[WorkingwithExternalFilesandProcesses-Syntax]]
+==== Syntax
+
+The serialization format supported by this parser is as follows:
+
+.Serialization format
+[source,text]
+----
+content ::= version (stored)? tokens
+version ::= digit+ " "
+; stored field value - any "=" inside must be escaped!
+stored ::= "=" text "="
+tokens ::= (token ((" ") + token)*)*
+token ::= text ("," attrib)*
+attrib ::= name '=' value
+name ::= text
+value ::= text
+----
+
+Special characters in "text" values can be escaped using the escape character `\` . The following escape sequences are recognized:
+
+[width="30%",options="header",]
+|===
+|EscapeSequence |Description
+|"`\` " |literal space character
+|"`\,`" |literal `,` character
+|"`\=`" |literal `=` character
+|"`\\`" |literal `\` character
+|"`\n`" |newline
+|"`\r`" |carriage return
+|"`\t`" |horizontal tab
+|===
+
+Please note that Unicode sequences (e.g. `\u0001`) are not supported.
+
+[[WorkingwithExternalFilesandProcesses-Supportedattributenames]]
+==== Supported attribute names
+
+The following token attributes are supported, and identified with short symbolic names:
+
+// TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
+
+[cols="10,30,30,30",options="header"]
+|===
+|Name |Description |Lucene attribute |Value format
+|`i` |position increment |{lucene-javadocs}/core/org/apache/lucene/analysis/tokenattributes/PositionIncrementAttribute.html[PositionIncrementAttribute] |integer
+|`s` |start offset |{lucene-javadocs}/core/org/apache/lucene/analysis/tokenattributes/OffsetAttribute.html[OffsetAttribute] |integer
+|`e` |end offset |OffsetAttribute |integer
+|`y` |lexical type |{lucene-javadocs}/core/org/apache/lucene/analysis/tokenattributes/TypeAttribute.html[TypeAttribute] |string
+|`f` |flags |{lucene-javadocs}/core/org/apache/lucene/analysis/tokenattributes/FlagsAttribute.html[FlagsAttribute] |hexadecimal integer
+|`p` |payload |{lucene-javadocs}/core/org/apache/lucene/analysis/tokenattributes/PayloadAttribute.html[PayloadAttribute] |bytes in hexadecimal format; whitespace is ignored
+|===
+
+Token positions are tracked and implicitly added to the token stream - the start and end offsets consider only the term text and whitespace, and exclude the space taken by token attributes.
+
+[[WorkingwithExternalFilesandProcesses-Exampletokenstreams]]
+==== Example token streams
+
+// TODO: in cwiki each of these examples was in it's own "panel" ... do we want something like that here?
+// TODO: these examples match what was in cwiki, but I'm honestly not sure if the formatting there was correct to start?
+
+[source,text]
+----
+1 one two three
+----
+
+* version: 1
+* stored: null
+* token: (term=`one`,startOffset=0,endOffset=3)
+* token: (term=`two`,startOffset=4,endOffset=7)
+* token: (term=`three`,startOffset=8,endOffset=13)
+
+[source,text]
+----
+1 one two three
+----
+
+* version: 1
+* stored: null
+* token: (term=`one`,startOffset=0,endOffset=3)
+* token: (term=`two`,startOffset=5,endOffset=8)
+* token: (term=`three`,startOffset=11,endOffset=16)
+
+[source,text]
+----
+1 one,s=123,e=128,i=22 two three,s=20,e=22
+----
+
+* version: 1
+* stored: null
+* token: (term=`one`,positionIncrement=22,startOffset=123,endOffset=128)
+* token: (term=`two`,positionIncrement=1,startOffset=5,endOffset=8)
+* token: (term=three,positionIncrement=1,startOffset=20,endOffset=22)
+
+[source,text]
+----
+1 \ one\ \,,i=22,a=\, two\=
+
+\n,\ =\ \
+----
+
+* version: 1
+* stored: null
+* token: (term=` one ,`,positionIncrement=22,startOffset=0,endOffset=6)
+* token: (term=`two=` ,positionIncrement=1,startOffset=7,endOffset=15)
+* token: (term=`\`,positionIncrement=1,startOffset=17,endOffset=18)
+
+Note that unknown attributes and their values are ignored, so in this example, the "```a```" attribute on the first token and the " " (escaped space) attribute on the second token are ignored, along with their values, because they are not among the supported attribute names.
+
+[source,text]
+----
+1 ,i=22 ,i=33,s=2,e=20 ,
+----
+
+* version: 1
+* stored: null
+* token: (term=,positionIncrement=22,startOffset=0,endOffset=0)
+* token: (term=,positionIncrement=33,startOffset=2,endOffset=20)
+* token: (term=,positionIncrement=1,startOffset=2,endOffset=2)
+
+[source,text]
+----
+1 =This is the stored part with \=
+\n \t escapes.=one two three
+----
+
+* version: 1
+* stored: "`This is the stored part with = \t escapes.`"
+* token: (term=`one`,startOffset=0,endOffset=3)
+* token: (term=`two`,startOffset=4,endOffset=7)
+* token: (term=`three`,startOffset=8,endOffset=13)
+
+Note that the "`\t`" in the above stored value is not literal; it's shown that way to visually indicate the actual tab char that is in the stored value.
+
+[source,text]
+----
+1 ==
+----
+
+* version: 1
+* stored: ""
+* (no tokens)
+
+[source,text]
+----
+1 =this is a test.=
+----
+
+* version: 1
+* stored: "this is a test."
+* (no tokens)
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/95968c69/solr/solr-ref-guide/src/zookeeper-access-control.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/zookeeper-access-control.adoc b/solr/solr-ref-guide/src/zookeeper-access-control.adoc
new file mode 100644
index 0000000..c0fa805
--- /dev/null
+++ b/solr/solr-ref-guide/src/zookeeper-access-control.adoc
@@ -0,0 +1,148 @@
+= ZooKeeper Access Control
+:page-shortname: zookeeper-access-control
+:page-permalink: zookeeper-access-control.html
+
+This section describes using ZooKeeper access control lists (ACLs) with Solr. For information about ZooKeeper ACLs, see the ZooKeeper documentation at http://zookeeper.apache.org/doc/r3.4.6/zookeeperProgrammers.html#sc_ZooKeeperAccessControl.
+
+[[ZooKeeperAccessControl-AboutZooKeeperACLs]]
+== About ZooKeeper ACLs
+
+SolrCloud uses ZooKeeper for shared information and for coordination.
+
+This section describes how to configure Solr to add more restrictive ACLs to the ZooKeeper content it creates, and how to tell Solr about the credentials required to access the content in ZooKeeper. If you want to use ACLs in your ZooKeeper nodes, you will have to activate this functionality; by default, Solr behavior is open-unsafe ACL everywhere and uses no credentials.
+
+Content stored in ZooKeeper is critical to the operation of a SolrCloud cluster. Open access to SolrCloud content on ZooKeeper could lead to a variety of problems. For example:
+
+* Changing configuration might cause Solr to fail or behave in an unintended way.
+* Changing cluster state information into something wrong or inconsistent might very well make a SolrCloud cluster behave strangely.
+* Adding a delete-collection job to be carried out by the Overseer will cause data to be deleted from the cluster.
+
+You may want to enable ZooKeeper ACLs with Solr if you grant access to your ZooKeeper ensemble to entities you do not trust, or if you want to reduce risk of bad actions resulting from, e.g.:
+
+* Malware that found its way into your system.
+* Other systems using the same ZooKeeper ensemble (a "bad thing" might be done by accident).
+
+You might even want to limit read-access, if you think there is stuff in ZooKeeper that not everyone should know about. Or you might just in general work on a need-to-know basis.
+
+Protecting ZooKeeper itself could mean many different things. **This section is about protecting Solr content in ZooKeeper**. ZooKeeper content basically lives persisted on disk and (partly) in memory of the ZooKeeper processes. *This section is not about protecting ZooKeeper data at storage or ZooKeeper process levels* - that's for ZooKeeper to deal with.
+
+But this content is also available to "the outside" via the ZooKeeper API. Outside processes can connect to ZooKeeper and create/update/delete/read content; for example, a Solr node in a SolrCloud cluster wants to create/update/delete/read, and a SolrJ client wants to read from the cluster. It is the responsibility of the outside processes that create/update content to setup ACLs on the content. ACLs describe who is allowed to read, update, delete, create, etc. Each piece of information (znode/content) in ZooKeeper has its own set of ACLs, and inheritance or sharing is not possible. The default behavior in Solr is to add one ACL on all the content it creates - one ACL that gives anyone the permission to do anything (in ZooKeeper terms this is called "the open-unsafe ACL").
+
+[[ZooKeeperAccessControl-HowtoEnableACLs]]
+== How to Enable ACLs
+
+We want to be able to:
+
+. Control the credentials Solr uses for its ZooKeeper connections. The credentials are used to get permission to perform operations in ZooKeeper.
+. Control which ACLs Solr will add to znodes (ZooKeeper files/folders) it creates in ZooKeeper.
+. Control it "from the outside", so that you do not have to modify and/or recompile Solr code to turn this on.
+
+Solr nodes, clients and tools (e.g. ZkCLI) always use a java class called {solr-javadocs}/solr-solrj/org/apache/solr/common/cloud/SolrZkClient.html[`SolrZkClient`] to deal with their ZooKeeper stuff. The implementation of the solution described here is all about changing `SolrZkClient`. If you use `SolrZkClient` in your application, the descriptions below will be true for your application too.
+
+[[ZooKeeperAccessControl-ControllingCredentials]]
+=== Controlling Credentials
+
+You control which credentials provider will be used by configuring the `zkCredentialsProvider` property in `solr.xml`'s `<solrcloud>` section to the name of a class (on the classpath) implementing the {solr-javadocs}/solr-solrj/org/apache/solr/common/cloud/ZkCredentialsProvider[`ZkCredentialsProvider`] interface. `server/solr/solr.xml` in the Solr distribution defines the `zkCredentialsProvider` such that it will take on the value of the same-named `zkCredentialsProvider` system property if it is defined (e.g. by uncommenting the `SOLR_ZK_CREDS_AND_ACLS` environment variable definition in `solr.in.sh/.cmd` - see below), or if not, default to the `DefaultZkCredentialsProvider` implementation.
+
+*Out of the Box Implementations*
+
+You can always make you own implementation, but Solr comes with two implementations:
+
+* `org.apache.solr.common.cloud.DefaultZkCredentialsProvider`: Its `getCredentials()` returns a list of length zero, or "no credentials used". This is the default.
+* `org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider`: This lets you define your credentials using system properties. It supports at most one set of credentials.
+** The schema is "digest". The username and password are defined by system properties "```zkDigestUsername```" and "```zkDigestPassword```", respectively. This set of credentials will be added to the list of credentials returned by `getCredentials()` if both username and password are provided.
+** If the one set of credentials above is not added to the list, this implementation will fall back to default behavior and use the (empty) credentials list from `DefaultZkCredentialsProvider`.
+
+[[ZooKeeperAccessControl-ControllingACLs]]
+=== Controlling ACLs
+
+You control which ACLs will be added by configuring `zkACLProvider` property in `solr.xml`'s `<solrcloud>` section to the name of a class (on the classpath) implementing the {solr-javadocs}//solr-solrj/org/apache/solr/common/cloud/ZkACLProvider[`ZkACLProvider`] interface. `server/solr/solr.xml` in the Solr distribution defines the `zkACLProvider` such that it will take on the value of the same-named `zkACLProvider` system property if it is defined (e.g. by uncommenting the `SOLR_ZK_CREDS_AND_ACLS` environment variable definition in `solr.in.sh/.cmd` - see below), or if not, default to the `DefaultZkACLProvider` implementation.
+
+[[ZooKeeperAccessControl-OutoftheBoxImplementations]]
+==== Out of the Box Implementations
+
+You can always make you own implementation, but Solr comes with:
+
+* `org.apache.solr.common.cloud.DefaultZkACLProvider`: It returns a list of length one for all `zNodePath`-s. The single ACL entry in the list is "open-unsafe". This is the default.
+* `org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider`: This lets you define your ACLs using system properties. Its `getACLsToAdd()` implementation does not use `zNodePath` for anything, so all znodes will get the same set of ACLs. It supports adding one or both of these options:
+** A user that is allowed to do everything.
+*** The permission is "```ALL```" (corresponding to all of `CREATE`, `READ`, `WRITE`, `DELETE`, and `ADMIN`), and the schema is "digest".
+*** The username and password are defined by system properties "```zkDigestUsername```" and "```zkDigestPassword```", respectively.
+*** This ACL will not be added to the list of ACLs unless both username and password are provided.
+** A user that is only allowed to perform read operations.
+*** The permission is "```READ```" and the schema is "digest".
+*** The username and password are defined by system properties "```zkDigestReadonlyUsername```" and "```zkDigestReadonlyPassword```", respectively.
+*** This ACL will not be added to the list of ACLs unless both username and password are provided.
+* `org.apache.solr.common.cloud.SaslZkACLProvider`: Requires SASL authentication. Gives all permissions for the user specified in system property `solr.authorization.superuser` (default: `solr`) when using SASL, and gives read permissions for anyone else. Designed for a setup where configurations have already been set up and will not be modified, or where configuration changes are controlled via Solr APIs. This provider will be useful for administration in a kerberos environment. In such an environment, the administrator wants Solr to authenticate to ZooKeeper using SASL, since this is only way to authenticate with ZooKeeper via Kerberos.
+
+If none of the above ACLs is added to the list, the (empty) ACL list of `DefaultZkACLProvider` will be used by default.
+
+Notice the overlap in system property names with credentials provider `VMParamsSingleSetCredentialsDigestZkCredentialsProvider` (described above). This is to let the two providers collaborate in a nice and perhaps common way: we always protect access to content by limiting to two users - an admin-user and a readonly-user - AND we always connect with credentials corresponding to this same admin-user, basically so that we can do anything to the content/znodes we create ourselves.
+
+You can give the readonly credentials to "clients" of your SolrCloud cluster - e.g. to be used by SolrJ clients. They will be able to read whatever is necessary to run a functioning SolrJ client, but they will not be able to modify any content in ZooKeeper.
+
+
+[[ZooKeeperAccessControl-bin_solr_solr.cmd_server_scripts_cloud-scripts_zkcli.sh_zkcli.bat]]
+=== ZooKeeper ACLs in Solr Scripts
+
+There are two scripts that impact ZooKeeper ACLs:
+* For *nix systems: `bin/solr` & `server/scripts/cloud-scripts/zkcli.sh`
+* For Windows systems: `bin/solr.cmd` & `server/scripts/cloud-scripts/zkcli.bat`
+
+These Solr scripts can enable use of ZK ACLs by setting the appropriate system properties: uncomment the following and replace the passwords with ones you choose to enable the above-described VM parameters ACL and credentials providers in the following files:
+
+.solr.in.sh
+[source,bash]
+----
+# Settings for ZK ACL
+#SOLR_ZK_CREDS_AND_ACLS="-DzkACLProvider=org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider \
+# -DzkCredentialsProvider=org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider \
+# -DzkDigestUsername=admin-user -DzkDigestPassword=CHANGEME-ADMIN-PASSWORD \
+# -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD"
+#SOLR_OPTS="$SOLR_OPTS $SOLR_ZK_CREDS_AND_ACLS"
+----
+
+.solr.in.cmd
+[source,powershell]
+----
+REM Settings for ZK ACL
+REM set SOLR_ZK_CREDS_AND_ACLS=-DzkACLProvider=org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider ^
+REM -DzkCredentialsProvider=org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider ^
+REM -DzkDigestUsername=admin-user -DzkDigestPassword=CHANGEME-ADMIN-PASSWORD ^
+REM -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD
+REM set SOLR_OPTS=%SOLR_OPTS% %SOLR_ZK_CREDS_AND_ACLS%
+----
+
+.zkcli.sh
+[source,bash]
+----
+# Settings for ZK ACL
+#SOLR_ZK_CREDS_AND_ACLS="-DzkACLProvider=org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider \
+# -DzkCredentialsProvider=org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider \
+# -DzkDigestUsername=admin-user -DzkDigestPassword=CHANGEME-ADMIN-PASSWORD \
+# -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD"
+----
+
+.zkcli.bat
+[source,powershell]
+----
+REM Settings for ZK ACL
+REM set SOLR_ZK_CREDS_AND_ACLS=-DzkACLProvider=org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider ^
+REM -DzkCredentialsProvider=org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider ^
+REM -DzkDigestUsername=admin-user -DzkDigestPassword=CHANGEME-ADMIN-PASSWORD ^
+REM -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD
+----
+
+[[ZooKeeperAccessControl-ChangingACLSchemes]]
+== Changing ACL Schemes
+
+Over the lifetime of operating your Solr cluster, you may decide to move from an unsecured ZooKeeper to a secured instance. Changing the configured `zkACLProvider` in `solr.xml` will ensure that newly created nodes are secure, but will not protect the already existing data. To modify all existing ACLs, you can use the `updateacls` command with Solr's ZkCLI. First uncomment the `SOLR_ZK_CREDS_AND_ACLS` environment variable definition in `server/scripts/cloud-scripts/zkcli.sh` (or `zkcli.bat` on Windows) and fill in the passwords for the admin-user and the readonly-user - see above - then run `server/scripts/cloud-scripts/zkcli.sh -cmd updateacls /zk-path`, or on Windows run `server\scripts\cloud-scripts\zkcli.bat cmd updateacls /zk-path`.
+
+Changing ACLs in ZK should only be done while your SolrCloud cluster is stopped. Attempting to do so while Solr is running may result in inconsistent state and some nodes becoming inaccessible.
+
+The VM properties `zkACLProvider` and `zkCredentialsProvider`, included in the `SOLR_ZK_CREDS_AND_ACLS` environment variable in `zkcli.sh/.bat`, control the conversion:
+
+* The Credentials Provider must be one that has current admin privileges on the nodes. When omitted, the process will use no credentials (suitable for an unsecure configuration).
+* The ACL Provider will be used to compute the new ACLs. When omitted, the process will set all permissions to all users, removing any security present.
+
+The uncommented `SOLR_ZK_CREDS_AND_ACLS` environment variable in `zkcli.sh/.bat` sets the credentials and ACL providers to the `VMParamsSingleSetCredentialsDigestZkCredentialsProvider` and `VMParamsAllAndReadonlyDigestZkACLProvider` implementations, described earlier in the page.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/95968c69/solr/solr-ref-guide/tools/BuildNavAndPDFBody.java
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/tools/BuildNavAndPDFBody.java b/solr/solr-ref-guide/tools/BuildNavAndPDFBody.java
new file mode 100644
index 0000000..75638a9
--- /dev/null
+++ b/solr/solr-ref-guide/tools/BuildNavAndPDFBody.java
@@ -0,0 +1,283 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.io.*;
+import java.io.FilenameFilter;
+import java.util.*;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.json.*;
+
+import org.asciidoctor.Asciidoctor.Factory;
+import org.asciidoctor.Asciidoctor;
+import org.asciidoctor.ast.DocumentHeader;
+
+
+public class BuildNavAndPDFBody {
+
+ public static void main(String[] args) throws Exception {
+ if (args.length != 2) {
+ throw new RuntimeException("Wrong # of args: " + args.length);
+ }
+
+ final File adocDir = new File(args[0]);
+ final String mainPageShortname = args[1];
+ if (! adocDir.exists()) {
+ throw new RuntimeException("asciidoc directory does not exist: " + adocDir.toString());
+ }
+
+ // build up a quick mapping of every known page
+ System.out.println("Building up tree of all known pages");
+ final Map<String,Page> allPages = new LinkedHashMap<String,Page>();
+ Asciidoctor doctor = null;
+ try {
+ doctor = Factory.create();
+ final File[] adocFiles = adocDir.listFiles(ADOC_FILE_NAMES);
+ for (File file : adocFiles) {
+ Page page = new Page(file, doctor.readDocumentHeader(file));
+ if (allPages.containsKey(page.shortname)) {
+ throw new RuntimeException("multiple pages with same shortname: " + page.file.toString() + " and " + allPages.get(page.shortname));
+ }
+ allPages.put(page.shortname, page);
+ }
+ } finally {
+ if (null != doctor) {
+ doctor.shutdown();
+ doctor = null;
+ }
+ }
+
+ // build up a hierarchical structure rooted at our mainPage
+ final Page mainPage = allPages.get(mainPageShortname);
+ if (null == mainPage) {
+ throw new RuntimeException("no main-page found with shortname: " + mainPageShortname);
+ }
+ mainPage.buildKidsRecursive(allPages);
+
+ // TODO: use depthFirstWalk to prune allPages to validate that we don't have any loops or orphan pages
+
+
+ // Build up the PDF file,
+ // while doing this also build up some next/prev maps for use in building the scrollnav
+ File pdfFile = new File(new File(adocDir, "_data"), "pdf-main-body.adoc");
+ if (pdfFile.exists()) {
+ throw new RuntimeException(pdfFile.toString() + " already exists");
+ }
+ final Map<String,Page> nextPage = new HashMap<String,Page>();
+ final Map<String,Page> prevPage = new HashMap<String,Page>();
+ System.out.println("Creating " + pdfFile.toString());
+ try (Writer w = new OutputStreamWriter(new FileOutputStream(pdfFile), "UTF-8")) {
+ // Note: not worrying about headers or anything like that ...
+ // expecting this file to just be included by the main PDF file.
+
+ // track how deep we are so we can adjust headers accordingly
+ // start with a "negative" depth to treat all "top level" pages as same depth as main-page using Math.max
+ // (see below)
+ final AtomicInteger depth = new AtomicInteger(-1);
+
+ // the previous page seen in our walk
+ AtomicReference<Page> previous = new AtomicReference<Page>();
+
+ mainPage.depthFirstWalk(new Page.RecursiveAction() {
+ public boolean act(Page page) {
+ try {
+ if (null != previous.get()) {
+ // add previous as our 'prev' page, and ourselves as the 'next' of previous
+ prevPage.put(page.shortname, previous.get());
+ nextPage.put(previous.get().shortname, page);
+ }
+ previous.set(page);
+
+
+ // HACK: where this file actually lives will determine what we need here...
+ w.write("include::../");
+ w.write(page.file.getName());
+ w.write("[leveloffset=+"+Math.max(0, depth.intValue())+"]\n\n");
+ depth.incrementAndGet();
+ return true;
+ } catch (IOException ioe) {
+ throw new RuntimeException("IOE recursively acting on " + page.shortname, ioe);
+ }
+ }
+ public void postKids(Page page) {
+ depth.decrementAndGet();
+ }
+ });
+ }
+
+ // Build up the scrollnav file for jekyll's footer
+ File scrollnavFile = new File(new File(adocDir, "_data"), "scrollnav.json");
+ if (scrollnavFile.exists()) {
+ throw new RuntimeException(scrollnavFile.toString() + " already exists");
+ }
+ System.out.println("Creating " + scrollnavFile.toString());
+ try (Writer w = new OutputStreamWriter(new FileOutputStream(scrollnavFile), "UTF-8")) {
+ JSONObject scrollnav = new JSONObject();
+ for (Page p : allPages.values()) {
+ JSONObject current = new JSONObject();
+ Page prev = prevPage.get(p.shortname);
+ Page next = nextPage.get(p.shortname);
+ if (null != prev) {
+ current.put("prev",
+ new JSONObject()
+ .put("url", prev.permalink)
+ .put("title", prev.title));
+ }
+ if (null != next) {
+ current.put("next",
+ new JSONObject()
+ .put("url", next.permalink)
+ .put("title", next.title));
+ }
+ scrollnav.put(p.shortname, current);
+ }
+ // HACK: jekyll doesn't like escaped forward slashes in it's JSON?
+ w.write(scrollnav.toString(2).replaceAll("\\\\/","/"));
+ }
+
+ // Build up the sidebar file for jekyll
+ File sidebarFile = new File(new File(adocDir, "_data"), "sidebar.json");
+ if (sidebarFile.exists()) {
+ throw new RuntimeException(sidebarFile.toString() + " already exists");
+ }
+ System.out.println("Creating " + sidebarFile.toString());
+ try (Writer w = new OutputStreamWriter(new FileOutputStream(sidebarFile), "UTF-8")) {
+ // A stack for tracking what we're working on as we recurse
+ final Stack<JSONObject> stack = new Stack<JSONObject>();
+
+ mainPage.depthFirstWalk(new Page.RecursiveAction() {
+ public boolean act(Page page) {
+ final int depth = stack.size();
+ if (4 < depth) {
+ System.err.println("ERROR: depth==" + depth + " for " + page.permalink);
+ System.err.println("sidebar.html template can not support pages this deep");
+ System.exit(-1);
+ }
+ try {
+ final JSONObject current = new JSONObject()
+ .put("title",page.title)
+ .put("url", page.permalink)
+ .put("depth", depth)
+ .put("kids", new JSONArray());
+
+ if (0 < depth) {
+ JSONObject parent = stack.peek();
+ ((JSONArray)parent.get("kids")).put(current);
+ }
+
+ stack.push(current);
+ } catch (JSONException e) {
+ throw new RuntimeException(e);
+ }
+ return true;
+ }
+ public void postKids(Page page) {
+ final JSONObject current = stack.pop();
+ if (0 == stack.size()) {
+ assert page == mainPage;
+ try {
+ // HACK: jekyll doesn't like escaped forward slashes in it's JSON?
+ w.write(current.toString(2).replaceAll("\\\\/","/"));
+ } catch (IOException | JSONException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+ });
+ }
+
+ }
+
+ /** Simple struct for modeling the key metadata for dealing with page navigation */
+ public static final class Page {
+ public final File file;
+ public final String title;
+ public final String shortname;
+ public final String permalink;
+ public final List<String> kidShortnames;
+ /** NOTE: not populated on construction
+ * @see #buildKidsRecursive
+ */
+ public final List<Page> kids;
+ private final List<Page> mutableKids;
+ public Page(File file, DocumentHeader header) {
+ this.file = file;
+ this.title = header.getDocumentTitle().getMain();
+
+ // TODO: do error checking if attribute metadata we care about is missing
+ Map<String,Object> attrs = header.getAttributes();
+ this.shortname = (String) attrs.get("page-shortname");
+ this.permalink = (String) attrs.get("page-permalink");
+
+ if (attrs.containsKey("page-children")) {
+ String kidsString = ((String) attrs.get("page-children")).trim();
+ this.kidShortnames = Collections.<String>unmodifiableList
+ (Arrays.asList(kidsString.split(",\\s+")));
+ this.mutableKids = new ArrayList<Page>(kidShortnames.size());
+ } else {
+ this.kidShortnames = Collections.<String>emptyList();
+ this.mutableKids = Collections.<Page>emptyList();
+ }
+ this.kids = Collections.<Page>unmodifiableList(mutableKids);
+ }
+
+ /** Recursively populates {@link #kids} from {@link #kidShortnames} via the <code>allPages</code> Map */
+ public void buildKidsRecursive(Map<String,Page> allPages) {
+ for (String kidShortname : kidShortnames) {
+ Page kid = allPages.get(kidShortname);
+ if (null == kid) {
+ throw new RuntimeException("Unable to locate " + kidShortname + "; child of " + shortname + "("+file.toString());
+ }
+ mutableKids.add(kid);
+ kid.buildKidsRecursive(allPages);
+ }
+ }
+
+ /**
+ * Do a depth first recursive action on this node and it's {@link #kids}
+ * @see RecursiveAction
+ */
+ public void depthFirstWalk(RecursiveAction action) {
+ if (action.act(this)) {
+ for (Page kid : kids) {
+ kid.depthFirstWalk(action);
+ }
+ action.postKids(this);
+ }
+ }
+
+ /** @see #depthFirstWalk */
+ public static interface RecursiveAction {
+ /** return true if kids should also be visited */
+ public boolean act(Page page);
+ /**
+ * called after recusion to each kid (if any) of specified node,
+ * never called if {@link #act} returned false
+ */
+ public default void postKids(Page page) { /* No-op */ };
+ }
+ }
+
+
+ /** Trivial filter for only "*.adoc" files */
+ public static final FilenameFilter ADOC_FILE_NAMES = new FilenameFilter() {
+ public boolean accept(File dir, String name) {
+ return name.endsWith(".adoc");
+ }
+ };
+}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/95968c69/solr/solr-ref-guide/tools/CheckLinksAndAnchors.java
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/tools/CheckLinksAndAnchors.java b/solr/solr-ref-guide/tools/CheckLinksAndAnchors.java
new file mode 100644
index 0000000..c5dcac2
--- /dev/null
+++ b/solr/solr-ref-guide/tools/CheckLinksAndAnchors.java
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.io.*;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.MalformedURLException;
+import java.nio.file.Files;
+import java.util.Arrays;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Locale;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.jsoup.Jsoup;
+import org.jsoup.nodes.Document;
+import org.jsoup.nodes.Element;
+import org.jsoup.nodes.Node;
+import org.jsoup.nodes.TextNode;
+import org.jsoup.parser.Parser;
+import org.jsoup.parser.Tag;
+import org.jsoup.select.Elements;
+import org.jsoup.select.NodeVisitor;
+
+/**
+ * Check various things regarding links in the generated HTML site.
+ * <p>
+ * Asciidoctor doesn't do a good job of rectifying situations where multiple documents are included in one
+ * massive (PDF) document may have identical anchors (either explicitly defined, or implicitly defined because of
+ * section headings). Asciidoctor also doesn't support linking directly to another (included) document by name,
+ * unless there is an explicit '#fragement' used inthe link.
+ * </p>
+ * <p>
+ * This tool parses the generated HTML site, looking for these situations in order to fail the build -- since the
+ * equivilent PDF will be broken. It also does sme general check of the relative URLs to ensure the destination
+ * files/anchors actaully exist.
+ * </p>
+ *
+ * TODO: build a list of all known external links so that some other tool could (optionally) ping them all for 200 status?
+ *
+ * @see https://github.com/asciidoctor/asciidoctor/issues/1865
+ * @see https://github.com/asciidoctor/asciidoctor/issues/1866
+ */
+public class CheckLinksAndAnchors {
+
+ public static final class HtmlFileFilter implements FileFilter {
+ public boolean accept(File pathname) {
+ return pathname.getName().toLowerCase().endsWith("html");
+ }
+ }
+
+ public static void main(String[] args) throws Exception {
+ int problems = 0;
+
+ if (args.length != 1) {
+ System.err.println("usage: CheckLinksAndAnchors <htmldir>");
+ System.exit(-1);
+ }
+ final File htmlDir = new File(args[0]);
+
+ final File[] pages = htmlDir.listFiles(new HtmlFileFilter());
+ if (0 == pages.length) {
+ System.err.println("No HTML Files found, wrong htmlDir? forgot to built the site?");
+ System.exit(-1);
+ }
+
+ final Map<String,List<File>> idsToFiles = new HashMap<>();
+ final Map<File,List<URI>> filesToRelativeLinks = new HashMap<>();
+ final Set<String> idsInMultiFiles = new HashSet<>(0);
+
+ for (File file : pages) {
+ //System.out.println("input File URI: " + file.toURI().toString());
+
+ assert ! filesToRelativeLinks.containsKey(file);
+ final List<URI> linksInThisFile = new ArrayList<URI>(17);
+ filesToRelativeLinks.put(file, linksInThisFile);
+
+ final String fileContents = readFile(file.getPath());
+ final Document doc = Jsoup.parse(fileContents);
+ // we only care about class='main-content' -- we don't want to worry
+ // about ids/links duplicated in the header/footer of every page,
+ final Element mainContent = doc.select(".main-content").first();
+ if (mainContent == null) {
+ throw new RuntimeException(file.getName() + " has no main-content div");
+ }
+
+ // Add all of the IDs in (the main-content of) this doc to idsToFiles (and idsInMultiFiles if needed)
+ final Elements nodesWithIds = mainContent.select("[id]");
+ // NOTE: add <body> to the nodesWithIds so we check the main section anchor as well
+ nodesWithIds.addAll(doc.select("body[id]"));
+ for (Element node : nodesWithIds) {
+ final String id = node.id();
+ assert null != id;
+ assert 0 != id.length();
+
+ // special case ids that we ignore
+ if (id.equals("preamble")) {
+ continue;
+ }
+
+ if (idsToFiles.containsKey(id)) {
+ idsInMultiFiles.add(id);
+ } else {
+ idsToFiles.put(id, new ArrayList<File>(1));
+ }
+ idsToFiles.get(id).add(file);
+ }
+
+ // check for (relative) links that don't include a fragment
+ final Elements links = mainContent.select("a[href]");
+ for (Element link : links) {
+ final String href = link.attr("href");
+ if (0 == href.length()) {
+ problems++;
+ System.err.println(file.toURI().toString() + " contains link with empty href");
+ }
+ try {
+ final URI uri = new URI(href);
+ if (! uri.isAbsolute()) {
+ final String frag = uri.getFragment();
+ if (null == frag || "".equals(frag)) {
+ // we must have a fragment for intra-page links to work correctly
+ problems++;
+ System.err.println(file.toURI().toString() + " contains relative link w/o an '#anchor': " + href);
+ } else {
+ // track the link to validate it exists in the target doc
+ linksInThisFile.add(uri);
+ }
+ }
+ } catch (URISyntaxException uri_ex) {
+ // before reporting a problem, see if it can be parsed as a valid (absolute) URL
+ // some solr examples URLs have characters that aren't legal URI characters
+ // Example: "ipod^3.0", "foo:[*+TO+*]", etc...
+ boolean href_is_valid_absolute_url = false;
+ try {
+ // if this isn't absolute, it will fail
+ final URL ignored = new URL(href);
+ href_is_valid_absolute_url = true;
+ } catch (MalformedURLException url_ex) {
+ problems++;
+ System.err.println(file.toURI().toString() + " contains link w/ invalid syntax: " + href);
+ System.err.println(" ... as URI: " + uri_ex.toString());
+ System.err.println(" ... as URL: " + url_ex.toString());
+ }
+ }
+ }
+ }
+
+ // if there are problematic ids, report them
+ for (String id : idsInMultiFiles) {
+ problems++;
+ System.err.println("ID occurs multiple times: " + id);
+ for (File file : idsToFiles.get(id)) {
+ System.err.println(" ... " + file.toURI().toString());
+ }
+ }
+
+ // check every (realtive) link in every file to ensure the frag exists in the target page
+ for (Map.Entry<File,List<URI>> entry : filesToRelativeLinks.entrySet()) {
+ final File source = entry.getKey();
+ for (URI link : entry.getValue()) {
+ final String path = (null == link.getPath() || "".equals(link.getPath())) ? source.getName() : link.getPath();
+ final String frag = link.getFragment();
+ if ( ! idsInMultiFiles.contains(frag) ) { // skip problematic dups already reported
+ final File dest = new File(htmlDir, path);
+ if ( ! dest.exists() ) {
+ problems++;
+ System.err.println("Relative link points at dest file that doesn't exist: " + link);
+ System.err.println(" ... source: " + source.toURI().toString());
+ } else if ( ( ! idsToFiles.containsKey(frag) ) || // no file contains this id, or...
+ // id exists, but not in linked file
+ ( ! idsToFiles.get(frag).get(0).getName().equals(path) )) {
+ problems++;
+ System.err.println("Relative link points at id that doesn't exist in dest: " + link);
+ System.err.println(" ... source: " + source.toURI().toString());
+ }
+ }
+ }
+ }
+
+
+ if (0 < problems) {
+ System.err.println("Total of " + problems + " problems found");
+ System.exit(-1);
+ }
+ }
+
+ static String readFile(String fileName) throws IOException {
+ InputStream in = new FileInputStream(fileName);
+ Reader reader = new InputStreamReader(in,"UTF-8");
+ BufferedReader br = new BufferedReader(reader);
+ try {
+ StringBuilder sb = new StringBuilder();
+ String line = br.readLine();
+ while (line != null) {
+ sb.append(line);
+ sb.append("\n");
+ line = br.readLine();
+ }
+ return sb.toString();
+ } finally {
+ br.close();
+ }
+ }
+
+}
+