You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by no...@apache.org on 2017/10/03 13:38:42 UTC
[01/50] [abbrv] lucene-solr:feature/autoscaling: Ref Guide: Update
meta-doc info on lexers for source blocks
Repository: lucene-solr
Updated Branches:
refs/heads/feature/autoscaling 942b6715b -> 2c873cd8c
Ref Guide: Update meta-doc info on lexers for source blocks
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/feec5c6f
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/feec5c6f
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/feec5c6f
Branch: refs/heads/feature/autoscaling
Commit: feec5c6f091433651046439fe079000126606b3b
Parents: 1b88435
Author: Cassandra Targett <ct...@apache.org>
Authored: Mon Sep 25 13:14:31 2017 -0500
Committer: Cassandra Targett <ct...@apache.org>
Committed: Mon Sep 25 13:14:31 2017 -0500
----------------------------------------------------------------------
solr/solr-ref-guide/src/meta-docs/asciidoc-syntax.adoc | 8 ++++++++
1 file changed, 8 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/feec5c6f/solr/solr-ref-guide/src/meta-docs/asciidoc-syntax.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/meta-docs/asciidoc-syntax.adoc b/solr/solr-ref-guide/src/meta-docs/asciidoc-syntax.adoc
index e0e5aec..05a7421 100644
--- a/solr/solr-ref-guide/src/meta-docs/asciidoc-syntax.adoc
+++ b/solr/solr-ref-guide/src/meta-docs/asciidoc-syntax.adoc
@@ -64,6 +64,14 @@ If your code block will include line breaks, put 4 hyphens (`----`) before and a
More info: http://asciidoctor.org/docs/user-manual/#source-code-blocks
+==== Source Block Syntax Highlighting
+
+The PDF and HTML outputs use Pygments to add syntax highlighting to code examples. This is done by adding the language of the code block after the `source`, as shown in the above example source block (`xml` in that case).
+
+Pygments has a long selection of lexers available. You can see the full list at http://pygments.org/docs/lexers. Use one of the valid short names to get syntax highlighting for that language.
+
+Ideally, we will have an appropriate lexer to use for all source blocks, but that's not possible. When in doubt, choose `text`, or leave it blank.
+
=== Block Titles
Titles can be added to most blocks (images, source blocks, tables, etc.) by simply prefacing the title with a period (`.`). For example, to add a title to the source block example above:
[46/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-10842: replace
quickstart.mdtext content with link to Ref Guide tutorial
Posted by no...@apache.org.
SOLR-10842: replace quickstart.mdtext content with link to Ref Guide tutorial
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/5b3a5152
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/5b3a5152
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/5b3a5152
Branch: refs/heads/feature/autoscaling
Commit: 5b3a5152bdf11758b5a02efb60568098fe825d45
Parents: 266cc82
Author: Cassandra Targett <ct...@apache.org>
Authored: Mon Oct 2 17:14:11 2017 -0500
Committer: Cassandra Targett <ct...@apache.org>
Committed: Mon Oct 2 17:14:11 2017 -0500
----------------------------------------------------------------------
solr/site/quickstart.mdtext | 614 +--------------------------------------
1 file changed, 4 insertions(+), 610 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5b3a5152/solr/site/quickstart.mdtext
----------------------------------------------------------------------
diff --git a/solr/site/quickstart.mdtext b/solr/site/quickstart.mdtext
index fcd9be0..92f593d 100644
--- a/solr/site/quickstart.mdtext
+++ b/solr/site/quickstart.mdtext
@@ -1,612 +1,6 @@
-# Solr Quick Start
-
-## Overview
-
-This document covers getting Solr up and running, ingesting a variety of data sources into multiple collections,
-and getting a feel for the Solr administrative and search interfaces.
-
-## Requirements
-
-To follow along with this tutorial, you will need...
-
-1. To meet the [system requirements](SYSTEM_REQUIREMENTS.html)
-2. An Apache Solr release ([download](http://lucene.apache.org/solr/downloads.html)). This tutorial was written using Apache Solr 6.2.0.
-
-## Getting Started
-
-Please run the browser showing this tutorial and the Solr server on the same machine so tutorial links will correctly
-point to your Solr server.
-
-Begin by unzipping the Solr release and changing your working directory to the subdirectory where Solr was installed.
-Note that the base directory name may vary with the version of Solr downloaded. For example, with a shell in UNIX,
-Cygwin, or MacOS:
-
- /:$ ls solr*
- solr-X.Y.Z.zip
- /:$ unzip -q solr-X.Y.Z.zip
- /:$ cd solr-X.Y.Z/
-
-Note that "X.Y.Z" will be replaced by an official Solr version (i.e. 6.4.3, 7.0.0, etc.)
-
-To launch Solr, run: `bin/solr start -e cloud -noprompt`
-
- /solr-X.Y.Z:$ bin/solr start -e cloud -noprompt
-
- Welcome to the SolrCloud example!
-
- Starting up 2 Solr nodes for your example SolrCloud cluster.
- ...
-
- Started Solr server on port 8983 (pid=8404). Happy searching!
- ...
-
- Started Solr server on port 7574 (pid=8549). Happy searching!
- ...
-
- SolrCloud example running, please visit http://localhost:8983/solr
-
- /solr-X.Y.Z:$ _
-
-You can see that the Solr is running by loading the Solr Admin UI in your web browser: <http://localhost:8983/solr/>.
-This is the main starting point for administering Solr.
-
-Solr will now be running two "nodes", one on port 7574 and one on port 8983. There is one collection created
-automatically, `gettingstarted`, a two shard collection, each with two replicas.
-The [Cloud tab](http://localhost:8983/solr/#/~cloud) in the Admin UI diagrams the collection nicely:
-
-<img alt="Solr Quick Start: SolrCloud diagram" style="width:800px" src="images/quickstart-solrcloud.png" />
-
-## Indexing Data
-
-Your Solr server is up and running, but it doesn't contain any data. The Solr install includes the `bin/post` tool in
-order to facilitate getting various types of documents easily into Solr from the start. We'll be
-using this tool for the indexing examples below.
-
-You'll need a command shell to run these examples, rooted in the Solr install directory; the shell from where you
-launched Solr works just fine.
-
-* NOTE: Currently the `bin/post` tool does not have a comparable Windows script, but the underlying Java program invoked
-is available. See the
-[Post Tool, Windows section](https://cwiki.apache.org/confluence/display/solr/Post+Tool#PostTool-Windows)
-for details.
-
-### Indexing a directory of "rich" files
-
-Let's first index local "rich" files including HTML, PDF, Microsoft Office formats (such as MS Word), plain text and
-many other formats. `bin/post` features the ability to crawl a directory of files, optionally recursively even,
-sending the raw content of each file into Solr for extraction and indexing. A Solr install includes a `docs/`
-subdirectory, so that makes a convenient set of (mostly) HTML files built-in to start with.
-
- bin/post -c gettingstarted docs/
-
-Here's what it'll look like:
-
- /solr-X.Y.Z:$ bin/post -c gettingstarted docs/
- java -classpath /solr-X.Y.Z/dist/solr-core-X.Y.Z.jar -Dauto=yes -Dc=gettingstarted -Ddata=files -Drecursive=yes org.apache.solr.util.SimplePostTool docs/
- SimplePostTool version 5.0.0
- Posting files to [base] url http://localhost:8983/solr/gettingstarted/update...
- Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
- Entering recursive mode, max depth=999, delay=0s
- Indexing directory docs (3 files, depth=0)
- POSTing file index.html (text/html) to [base]/extract
- POSTing file quickstart.html (text/html) to [base]/extract
- POSTing file SYSTEM_REQUIREMENTS.html (text/html) to [base]/extract
- Indexing directory docs/changes (1 files, depth=1)
- POSTing file Changes.html (text/html) to [base]/extract
- ...
- 4329 files indexed.
- COMMITting Solr index changes to http://localhost:8983/solr/gettingstarted/update...
- Time spent: 0:01:16.252
-
-The command-line breaks down as follows:
-
- * `-c gettingstarted`: name of the collection to index into
- * `docs/`: a relative path of the Solr install `docs/` directory
-
-You have now indexed thousands of documents into the `gettingstarted` collection in Solr and committed these changes.
-You can search for "solr" by loading the Admin UI [Query tab](http://localhost:8983/solr/#/gettingstarted/query),
-enter "solr" in the `q` param (replacing `*:*`, which matches all documents), and "Execute Query".
-See the [Searching](#searching) section below for more information.
-
-To index your own data, re-run the directory indexing command pointed to your own directory of documents. For example,
-on a Mac instead of `docs/` try `~/Documents/` or `~/Desktop/`! You may want to start from a clean, empty system
-again rather than have your content in addition to the Solr `docs/` directory; see the Cleanup section [below](#cleanup)
-for how to get back to a clean starting point.
-
-### Indexing Solr XML
-
-Solr supports indexing structured content in a variety of incoming formats. The historically predominant format for
-getting structured content into Solr has been
-[Solr XML](https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers#UploadingDatawithIndexHandlers-XMLFormattedIndexUpdates).
-Many Solr indexers have been coded to process domain content into Solr XML output, generally HTTP POSTed directly to
-Solr's `/update` endpoint.
-
-<a name="techproducts"></a>
-Solr's install includes a handful of Solr XML formatted files with example data (mostly mocked tech product data).
-NOTE: This tech product data has a more domain-specific configuration, including schema and browse UI. The `bin/solr`
-script includes built-in support for this by running `bin/solr start -e techproducts` which not only starts Solr but
-also then indexes this data too (be sure to `bin/solr stop -all` before trying it out).
-However, the example below assumes Solr was started with `bin/solr start -e cloud` to stay consistent with all examples
-on this page, and thus the collection used is "gettingstarted", not "techproducts".
-
-Using `bin/post`, index the example Solr XML files in `example/exampledocs/`:
-
- bin/post -c gettingstarted example/exampledocs/*.xml
-
-Here's what you'll see:
-
- /solr-X.Y.Z:$ bin/post -c gettingstarted example/exampledocs/*.xml
- java -classpath /solr-X.Y.Z/dist/solr-core-X.Y.Z.jar -Dauto=yes -Dc=gettingstarted -Ddata=files org.apache.solr.util.SimplePostTool example/exampledocs/gb18030-example.xml ...
- SimplePostTool version 5.0.0
- Posting files to [base] url http://localhost:8983/solr/gettingstarted/update...
- Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
- POSTing file gb18030-example.xml (application/xml) to [base]
- POSTing file hd.xml (application/xml) to [base]
- POSTing file ipod_other.xml (application/xml) to [base]
- POSTing file ipod_video.xml (application/xml) to [base]
- POSTing file manufacturers.xml (application/xml) to [base]
- POSTing file mem.xml (application/xml) to [base]
- POSTing file money.xml (application/xml) to [base]
- POSTing file monitor.xml (application/xml) to [base]
- POSTing file monitor2.xml (application/xml) to [base]
- POSTing file mp500.xml (application/xml) to [base]
- POSTing file sd500.xml (application/xml) to [base]
- POSTing file solr.xml (application/xml) to [base]
- POSTing file utf8-example.xml (application/xml) to [base]
- POSTing file vidcard.xml (application/xml) to [base]
- 14 files indexed.
- COMMITting Solr index changes to http://localhost:8983/solr/gettingstarted/update...
- Time spent: 0:00:02.077
-
-...and now you can search for all sorts of things using the default
-[Solr Query Syntax](https://cwiki.apache.org/confluence/display/solr/The+Standard+Query+Parser#TheStandardQueryParser-SpecifyingTermsfortheStandardQueryParser)
-(a superset of the Lucene query syntax)...
-
-NOTE:
-You can browse the documents indexed at <http://localhost:8983/solr/gettingstarted/browse>. The `/browse` UI allows
-getting a feel for how Solr's technical capabilities can be worked with in a familiar, though a bit rough and
-prototypical, interactive HTML view. (The `/browse` view defaults to assuming the `gettingstarted` schema and data
-are a catch-all mix of structured XML, JSON, CSV example data, and unstructured rich documents. Your own data may not
-look ideal at first, though the `/browse` templates are customizable.)
-
-### Indexing JSON
-
-Solr supports indexing JSON, either arbitrary structured JSON or "Solr JSON" (which is similar to Solr XML).
-
-Solr includes a small sample Solr JSON file to illustrate this capability. Again using `bin/post`, index the
-sample JSON file:
-
- bin/post -c gettingstarted example/exampledocs/books.json
-
-You'll see:
-
- /solr-X.Y.Z:$ bin/post -c gettingstarted example/exampledocs/books.json
- java -classpath /solr-X.Y.Z/dist/solr-core-X.Y.Z.jar -Dauto=yes -Dc=gettingstarted -Ddata=files org.apache.solr.util.SimplePostTool example/exampledocs/books.json
- SimplePostTool version 5.0.0
- Posting files to [base] url http://localhost:8983/solr/gettingstarted/update...
- Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
- POSTing file books.json (application/json) to [base]/json/docs
- 1 files indexed.
- COMMITting Solr index changes to http://localhost:8983/solr/gettingstarted/update...
- Time spent: 0:00:00.493
-
-For more information on indexing Solr JSON, see the Solr Reference Guide section
-[Solr-Style JSON](https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers#UploadingDatawithIndexHandlers-Solr-StyleJSON)
-
-To flatten (and/or split) and index arbitrary structured JSON, a topic beyond this quick start guide, check out
-[Transforming and Indexing Custom JSON data](https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers#UploadingDatawithIndexHandlers-TransformingandIndexingCustomJSON).
-
-### Indexing CSV (Comma/Column Separated Values)
-
-A great conduit of data into Solr is via CSV, especially when the documents are homogeneous by all having the
-same set of fields. CSV can be conveniently exported from a spreadsheet such as Excel, or exported from databases such
-as MySQL. When getting started with Solr, it can often be easiest to get your structured data into CSV format and then
-index that into Solr rather than a more sophisticated single step operation.
-
-Using `bin/post` index the included example CSV file:
-
- bin/post -c gettingstarted example/exampledocs/books.csv
-
-In your terminal you'll see:
-
- /solr-X.Y.Z:$ bin/post -c gettingstarted example/exampledocs/books.csv
- java -classpath /solr-X.Y.Z/dist/solr-core-X.Y.Z.jar -Dauto=yes -Dc=gettingstarted -Ddata=files org.apache.solr.util.SimplePostTool example/exampledocs/books.csv
- SimplePostTool version 5.0.0
- Posting files to [base] url http://localhost:8983/solr/gettingstarted/update...
- Entering auto mode. File endings considered are xml,json,jsonl,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log
- POSTing file books.csv (text/csv) to [base]
- 1 files indexed.
- COMMITting Solr index changes to http://localhost:8983/solr/gettingstarted/update...
- Time spent: 0:00:00.109
-
-For more information, see the Solr Reference Guide section
-[CSV Formatted Index Updates](https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers#UploadingDatawithIndexHandlers-CSVFormattedIndexUpdates)
-
-### Other indexing techniques
-
-* Import records from a database using the
-[Data Import Handler (DIH)](https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler).
-
-* Use [SolrJ](https://cwiki.apache.org/confluence/display/solr/Using+SolrJ) from JVM-based languages or
-other [Solr clients](https://cwiki.apache.org/confluence/display/solr/Client+APIs) to programmatically create documents
-to send to Solr.
-
-* Use the Admin UI [Documents tab](http://localhost:8983/solr/#/gettingstarted/documents) to paste in a document to be
-indexed, or select `Document Builder` from the `Document Type` dropdown to build a document one field at a time.
-Click on the `Submit Document` button below the form to index your document.
-
-***
-
-## Updating Data
-
-You may notice that even if you index content in this guide more than once, it does not duplicate the results found.
-This is because the example `schema.xml` specifies a "`uniqueKey`" field called "`id`". Whenever you POST commands to
-Solr to add a document with the same value for the `uniqueKey` as an existing document, it automatically replaces it
-for you. You can see that that has happened by looking at the values for `numDocs` and `maxDoc` in the core-specific
-Overview section of the Solr Admin UI.
-
-`numDocs` represents the number of searchable documents in the index (and will be larger than the number of XML, JSON,
-or CSV files since some files contained more than one document). The maxDoc value may be larger as the maxDoc count
-includes logically deleted documents that have not yet been physically removed from the index. You can re-post the
-sample files over and over again as much as you want and `numDocs` will never increase, because the new documents will
-constantly be replacing the old.
-
-Go ahead and edit any of the existing example data files, change some of the data, and re-run the SimplePostTool
-command. You'll see your changes reflected in subsequent searches.
-
-## Deleting Data
-
-You can delete data by POSTing a delete command to the update URL and specifying the value of the document's unique key
-field, or a query that matches multiple documents (be careful with that one!). Since these commands are smaller, we
-specify them right on the command line rather than reference a JSON or XML file.
-
-Execute the following command to delete a specific document:
-
- bin/post -c gettingstarted -d "<delete><id>SP2514N</id></delete>"
-
-
-## Searching
-
-Solr can be queried via REST clients, cURL, wget, Chrome POSTMAN, etc., as well as via the native clients available for
-many programming languages.
+<!-- As of 2 Oct 2017, this page now automatically redirects to
+ the Solr Reference Guide page solr/guide/solr-tutorial.html -->
-The Solr Admin UI includes a query builder interface - see the `gettingstarted` query tab at
-<http://localhost:8983/solr/#/gettingstarted/query>. If you click the `Execute Query` button without changing anything
-in the form, you'll get 10 documents in JSON format (`*:*` in the `q` param matches all documents):
-
-<img style="border:1px solid #ccc; width:800px" src="images/quickstart-query-screen.png" alt="Solr Quick Start: gettingstarted Query tab"/>
-
-The URL sent by the Admin UI to Solr is shown in light grey near the top right of the above screenshot - if you click on
-it, your browser will show you the raw response. To use cURL, give the same URL in quotes on the `curl` command line:
-
- curl "http://localhost:8983/solr/gettingstarted/select?q=*:*"
-
-
-### Basics
-
-#### Search for a single term
-
-To search for a term, give it as the `q` param value in the core-specific Solr Admin UI Query section, replace `*:*`
-with the term you want to find. To search for "foundation":
-
- curl "http://localhost:8983/solr/gettingstarted/select?q=foundation"
-
-You'll see:
-
- $ curl "http://localhost:8983/solr/gettingstarted/select?q=foundation"
- {
- "responseHeader":{
- "zkConnected":true,
- "status":0,
- "QTime":527,
- "params":{
- "q":"foundation",
- "indent":"true",
- "wt":"json"}},
- "response":{"numFound":4156,"start":0,"maxScore":0.10203234,"docs":[
- {
- "id":"0553293354",
- "cat":["book"],
- "name":["Foundation"],
- ...
-
-The response indicates that there are 4,156 hits (`"numFound":4156`), of which the first 10 were returned, since by
-default `start=0` and `rows=10`. You can specify these params to page through results, where `start` is the
-(zero-based) position of the first result to return, and `rows` is the page size.
-
-To restrict fields returned in the response, use the `fl` param, which takes a comma-separated list of field names.
-E.g. to only return the `id` field:
-
- curl "http://localhost:8983/solr/gettingstarted/select?q=foundation&fl=id"
-
-`q=foundation` matches nearly all of the docs we've indexed, since most of the files under `docs/` contain
-"The Apache Software Foundation". To restrict search to a particular field, use the syntax "`q=field:value`",
-e.g. to search for `Foundation` only in the `name` field:
-
- curl "http://localhost:8983/solr/gettingstarted/select?q=name:Foundation"
-
-The above request returns only one document (`"numFound":1`) - from the response:
-
- ...
- "response":{"numFound":1,"start":0,"maxScore":2.5902672,"docs":[
- {
- "id":"0553293354",
- "cat":["book"],
- "name":["Foundation"],
- ...
-
-#### Phrase search
-
-To search for a multi-term phrase, enclose it in double quotes: `q="multiple terms here"`. E.g. to search for
-"CAS latency" - note that the space between terms must be converted to "`+`" in a URL (the Admin UI will handle URL
-encoding for you automatically):
-
- curl "http://localhost:8983/solr/gettingstarted/select?indent=true&q=\"CAS+latency\""
-
-You'll get back:
-
- {
- "responseHeader":{
- "zkConnected":true,
- "status":0,
- "QTime":391,
- "params":{
- "q":"\"CAS latency\"",
- "indent":"true",
- "wt":"json"}},
- "response":{"numFound":3,"start":0,"maxScore":22.027056,"docs":[
- {
- "id":"TWINX2048-3200PRO",
- "name":["CORSAIR XMS 2GB (2 x 1GB) 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) Dual Channel Kit System Memory - Retail"],
- "manu":["Corsair Microsystems Inc."],
- "manu_id_s":"corsair",
- "cat":["electronics", "memory"],
- "features":["CAS latency 2, 2-3-3-6 timing, 2.75v, unbuffered, heat-spreader"],
- ...
-
-#### Combining searches
-
-By default, when you search for multiple terms and/or phrases in a single query, Solr will only require that one of them
-is present in order for a document to match. Documents containing more terms will be sorted higher in the results list.
-
-You can require that a term or phrase is present by prefixing it with a "`+`"; conversely, to disallow the presence of a
-term or phrase, prefix it with a "`-`".
-
-To find documents that contain both terms "`one`" and "`three`", enter `+one +three` in the `q` param in the
-Admin UI Query tab. Because the "`+`" character has a reserved purpose in URLs (encoding the space character),
-you must URL encode it for `curl` as "`%2B`":
-
- curl "http://localhost:8983/solr/gettingstarted/select?q=%2Bone+%2Bthree"
-
-To search for documents that contain the term "`two`" but **don't** contain the term "`one`", enter `+two -one` in the
-`q` param in the Admin UI. Again, URL encode "`+`" as "`%2B`":
-
- curl "http://localhost:8983/solr/gettingstarted/select?q=%2Btwo+-one"
-
-#### In depth
-
-For more Solr search options, see the Solr Reference Guide's
-[Searching](https://cwiki.apache.org/confluence/display/solr/Searching) section.
-
-
-### Faceting
-
-One of Solr's most popular features is faceting. Faceting allows the search results to be arranged into subsets (or
-buckets or categories), providing a count for each subset. There are several types of faceting: field values, numeric
-and date ranges, pivots (decision tree), and arbitrary query faceting.
-
-#### Field facets
-
-In addition to providing search results, a Solr query can return the number of documents that contain each unique value
-in the whole result set.
-
-From the core-specific Admin UI Query tab, if you check the "`facet`" checkbox, you'll see a few facet-related options
-appear:
-
-<img style="border:1px solid #ccc" src="images/quickstart-admin-ui-facet-options.png" alt="Solr Quick Start: Query tab facet options"/>
-
-To see facet counts from all documents (`q=*:*`): turn on faceting (`facet=true`), and specify the field to facet on via
-the `facet.field` param. If you only want facets, and no document contents, specify `rows=0`. The `curl` command below
-will return facet counts for the `manu_id_s` field:
-
- curl 'http://localhost:8983/solr/gettingstarted/select?q=*:*&rows=0'\
- '&facet=true&facet.field=manu_id_s'
-
-In your terminal, you'll see:
-
- {
- "responseHeader":{
- "zkConnected":true,
- "status":0,
- "QTime":201,
- "params":{
- "q":"*:*",
- "facet.field":"manu_id_s",
- "indent":"true",
- "rows":"0",
- "wt":"json",
- "facet":"true"}},
- "response":{"numFound":4374,"start":0,"maxScore":1.0,"docs":[]
- },
- "facet_counts":{
- "facet_queries":{},
- "facet_fields":{
- "manu_id_s":[
- "corsair",3,
- "belkin",2,
- "canon",2,
- "apple",1,
- "asus",1,
- "ati",1,
- "boa",1,
- "dell",1,
- "eu",1,
- "maxtor",1,
- "nor",1,
- "uk",1,
- "viewsonic",1,
- "samsung",0]},
- "facet_ranges":{},
- "facet_intervals":{},
- "facet_heatmaps":{}}}
-
-#### Range facets
-
-For numerics or dates, it's often desirable to partition the facet counts into ranges rather than discrete values.
-A prime example of numeric range faceting, using the example product data, is `price`. In the `/browse` UI, it looks
-like this:
-
-<img style="border:1px solid #ccc" src="images/quickstart-range-facet.png" alt="Solr Quick Start: Range facets"/>
-
-The data for these price range facets can be seen in JSON format with this command:
-
- curl 'http://localhost:8983/solr/gettingstarted/select?q=*:*&rows=0'\
- '&facet=true'\
- '&facet.range=price'\
- '&f.price.facet.range.start=0'\
- '&f.price.facet.range.end=600'\
- '&f.price.facet.range.gap=50'\
- '&facet.range.other=after'
-
-In your terminal you will see:
-
- {
- "responseHeader":{
- "zkConnected":true,
- "status":0,
- "QTime":248,
- "params":{
- "facet.range":"price",
- "q":"*:*",
- "f.price.facet.range.start":"0",
- "facet.range.other":"after",
- "indent":"on",
- "f.price.facet.range.gap":"50",
- "rows":"0",
- "wt":"json",
- "facet":"true",
- "f.price.facet.range.end":"600"}},
- "response":{"numFound":4374,"start":0,"maxScore":1.0,"docs":[]
- },
- "facet_counts":{
- "facet_queries":{},
- "facet_fields":{},
- "facet_ranges":{
- "price":{
- "counts":[
- "0.0",19,
- "50.0",1,
- "100.0",0,
- "150.0",2,
- "200.0",0,
- "250.0",1,
- "300.0",1,
- "350.0",2,
- "400.0",0,
- "450.0",1,
- "500.0",0,
- "550.0",0],
- "gap":50.0,
- "after":2,
- "start":0.0,
- "end":600.0}},
- "facet_intervals":{},
- "facet_heatmaps":{}}}
-
-#### Pivot facets
-
-Another faceting type is pivot facets, also known as "decision trees", allowing two or more fields to be nested for all
-the various possible combinations. Using the example technical product data, pivot facets can be used to see how many
-of the products in the "book" category (the `cat` field) are in stock or not in stock. Here's how to get at the raw
-data for this scenario:
-
- curl 'http://localhost:8983/solr/gettingstarted/select?q=*:*&rows=0&facet=on&facet.pivot=cat,inStock'
-
-This results in the following response (trimmed to just the book category output), which says out of 14 items in the
-"book" category, 12 are in stock and 2 are not in stock:
-
- ...
- "facet_pivot":{
- "cat,inStock":[{
- "field":"cat",
- "value":"book",
- "count":14,
- "pivot":[{
- "field":"inStock",
- "value":true,
- "count":12},
- {
- "field":"inStock",
- "value":false,
- "count":2}]},
- ...
-
-#### More faceting options
-
-For the full scoop on Solr faceting, visit the Solr Reference Guide's
-[Faceting](https://cwiki.apache.org/confluence/display/solr/Faceting) section.
-
-
-### Spatial
-
-Solr has sophisticated geospatial support, including searching within a specified distance range of a given location
-(or within a bounding box), sorting by distance, or even boosting results by the distance. Some of the example tech
-products documents in `example/exampledocs/*.xml` have locations associated with them to illustrate the spatial
-capabilities. To run the tech products example, see the [techproducts example section](#techproducts). Spatial queries
-can be combined with any other types of queries, such as in this example of querying for "ipod" within 10 kilometers
-from San Francisco:
-
-<img style="border:1px solid #ccc; width:800px" src="images/quickstart-spatial.png" alt="Solr Quick Start: spatial search"/>
-
-The URL to this example is
-<http://localhost:8983/solr/techproducts/browse?q=ipod&pt=37.7752%2C-122.4232&d=10&sfield=store&fq=%7B%21bbox%7D&queryOpts=spatial&queryOpts=spatial>,
-leveraging the `/browse` UI to show a map for each item and allow easy selection of the location to search near.
-
-To learn more about Solr's spatial capabilities, see the Solr Reference Guide's
-[Spatial Search](https://cwiki.apache.org/confluence/display/solr/Spatial+Search) section.
-
-## Wrapping up
-
-If you've run the full set of commands in this quick start guide you have done the following:
-
-* Launched Solr into SolrCloud mode, two nodes, two collections including shards and replicas
-* Indexed a directory of rich text files
-* Indexed Solr XML files
-* Indexed Solr JSON files
-* Indexed CSV content
-* Opened the admin console, used its query interface to get JSON formatted results
-* Opened the /browse interface to explore Solr's features in a more friendly and familiar interface
-
-Nice work! The script (see below) to run all of these items took under two minutes! (Your run time may vary, depending
-on your computer's power and resources available.)
-
-Here's a Unix script for convenient copying and pasting in order to run the key commands for this quick start guide:
-
- date
- bin/solr start -e cloud -noprompt
- open http://localhost:8983/solr
- bin/post -c gettingstarted docs/
- open http://localhost:8983/solr/gettingstarted/browse
- bin/post -c gettingstarted example/exampledocs/*.xml
- bin/post -c gettingstarted example/exampledocs/books.json
- bin/post -c gettingstarted example/exampledocs/books.csv
- bin/post -c gettingstarted -d "<delete><id>SP2514N</id></delete>"
- bin/solr healthcheck -c gettingstarted
- date
-
-## Cleanup
-
-As you work through this guide, you may want to stop Solr and reset the environment back to the starting point.
-The following command line will stop Solr and remove the directories for each of the two nodes that the start script
-created:
-
- bin/solr stop -all ; rm -Rf example/cloud/
-
-## Where to next?
-
-For more information on Solr, check out the following resources:
-
- * [Solr Reference Guide](https://cwiki.apache.org/confluence/display/solr/Apache+Solr+Reference+Guide) (ensure you
- match the version of the reference guide with your version of Solr)
- * See also additional [Resources](http://lucene.apache.org/solr/resources.html)
+# Solr Quick Start
+Please see the [Solr Tutorial](https://lucene.apache.org/solr/guide/solr-tutorial.html) or additional [Resources](http://lucene.apache.org/solr/resources.html).
[06/50] [abbrv] lucene-solr:feature/autoscaling: Ref Guide: rename
Streaming docs to fix broken intra-doc links in the PDF due to mismatching
anchor names
Posted by no...@apache.org.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9beafb61/solr/solr-ref-guide/src/stream-evaluators.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/stream-evaluators.adoc b/solr/solr-ref-guide/src/stream-evaluators.adoc
deleted file mode 100644
index 3599710..0000000
--- a/solr/solr-ref-guide/src/stream-evaluators.adoc
+++ /dev/null
@@ -1,966 +0,0 @@
-= Stream Evaluator Reference
-:page-shortname: stream-evaluators
-:page-permalink: stream-evaluators.html
-:page-tocclass: right
-:page-toclevels: 1
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-
-Stream evaluators are different then stream sources or stream decorators. Both
-stream sources and stream decorators return streams of tuples. Stream evaluators are more like
-a traditional function that evaluates its parameters and
-returns an result. That result can be a single value, array, map or other structure.
-
-Stream evaluators can be nested so that the output of an evaluator becomes the input
-for another evaluator.
-
-Stream evaluators can be called in different contexts. For example a stream evaluator
-can be called on its own or it can be called within the context of a streaming expression.
-
-== abs
-
-The `abs` function will return the absolute value of the provided single parameter. The `abs` function will fail to execute if the value is non-numeric. If a null value is found then null will be returned as the result.
-
-=== abs Parameters
-
-* `Field Name | Raw Number | Number Evaluator`
-
-=== abs Syntax
-
-The expressions below show the various ways in which you can use the `abs` evaluator. Only one parameter is accepted. Returns a numeric value.
-
-[source,text]
-----
-abs(1) // 1, not really a good use case for it
-abs(-1) // 1, not really a good use case for it
-abs(add(fieldA,fieldB)) // absolute value of fieldA + fieldB
-abs(fieldA) // absolute value of fieldA
-----
-
-== acos
-
-The `acos` function returns the trigonometric arccosine of a number.
-
-=== acos Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the arccosine of.
-
-=== acos Syntax
-
-[source,text]
-----
-acos(100.4) // returns the arccosine of 100.4
-acos(fieldA) // returns the arccosine for fieldA.
-if(gt(fieldA,fieldB),sin(fieldA),sin(fieldB)) // if fieldA > fieldB then return the arccosine of fieldA, else return the arccosine of fieldB
-----
-
-== add
-
-The `add` function will take 2 or more numeric values and add them together. The `add` function will fail to execute if any of the values are non-numeric. If a null value is found then null will be returned as the result.
-
-=== add Parameters
-
-* `Field Name | Raw Number | Number Evaluator`
-* `Field Name | Raw Number | Number Evaluator`
-* `......`
-* `Field Name | Raw Number | Number Evaluator`
-
-=== add Syntax
-
-The expressions below show the various ways in which you can use the `add` evaluator. The number and order of these parameters do not matter and is not limited except that at least two parameters are required. Returns a numeric value.
-
-[source,text]
-----
-add(1,2,3,4) // 1 + 2 + 3 + 4 == 10
-add(1,fieldA) // 1 + value of fieldA
-add(fieldA,1.4) // value of fieldA + 1.4
-add(fieldA,fieldB,fieldC) // value of fieldA + value of fieldB + value of fieldC
-add(fieldA,div(fieldA,fieldB)) // value of fieldA + (value of fieldA / value of fieldB)
-add(fieldA,if(gt(fieldA,fieldB),fieldA,fieldB)) // if fieldA > fieldB then fieldA + fieldA, else fieldA + fieldB
-----
-
-== analyze
-
-The `analyze` function analyzes text using a Lucene/Solr analyzer and returns a list of tokens
-emitted by the analyzer. The `analyze` function can be called on its own or within the
-`select` and `cartasianProduct` Streaming Expressions.
-
-=== analyze Parameters
-
-* `Field Name` | `Raw Text`: Either the field in a tuple or the raw text to be analyzed.
-* `Analyzer Field Name`: The field name of the analyzer to use to analyze the text.
-
-=== analyze Syntax
-
-The expressions below show the various ways in which you can use the `analyze` evaluator.
-
-* Analyze the raw text: `analyze("hello world", analyzerField)`
-* Analyze a text field within a `select` expression. This will annotate the tuples with output of the analyzer: `select(expr, analyze(textField, analyzerField) as outField)`
-* Analyze a text field with a `cartesianProduct` expression. This will stream each token emitted by the analyzer in its own tuple: `cartesianProduct(expr, analyze(textField, analyzer) as outField)`
-
-== and
-
-The `and` function will return the logical AND of at least 2 boolean parameters. The function will fail to execute if any parameters are non-boolean or null. Returns a boolean value.
-
-=== and Parameters
-
-* `Field Name | Raw Boolean | Boolean Evaluator`
-* `Field Name | Raw Boolean | Boolean Evaluator`
-* `......`
-* `Field Name | Raw Boolean | Boolean Evaluator`
-
-=== and Syntax
-
-The expressions below show the various ways in which you can use the `and` evaluator. At least two parameters are required, but there is no limit to how many you can use.
-
-[source,text]
-----
-and(true,fieldA) // true && fieldA
-and(fieldA,fieldB) // fieldA && fieldB
-and(or(fieldA,fieldB),fieldC) // (fieldA || fieldB) && fieldC
-and(fieldA,fieldB,fieldC,or(fieldD,fieldE),fieldF)
-----
-
-== anova
-
-The `anova` function calculates the https://en.wikipedia.org/wiki/Analysis_of_variance[analysis of variance] for two or more numeric arrays.
-
-=== anova Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array` ... (two or more)
-
-=== anova Syntax
-
-[source,text]
-anova(numericArray1, numericArray2) // calculates ANOVA for two numeric arrays
-anova(numericArray1, numericArray2, numericArray2) // calculates ANOVA for three numeric arrays
-
-== array
-
-The `array` function returns an array of numerics or other objects including other arrays.
-
-=== array Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric` | `array` ...
-
-=== array Syntax
-
-[source,text]
-array(1, 2, 3) // Array of numerics
-array(array(1,2,3), array(4,5,6)) // Array of arrays
-
-== asin
-The `asin` function returns the trigonometric arcsine of a number.
-
-=== asin Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the arcsine of.
-
-=== asin Syntax
-
-[source,text]
-----
-asin(100.4) // returns the sine of 100.4
-asine(fieldA) // returns the sine for fieldA.
-if(gt(fieldA,fieldB),asin(fieldA),asin(fieldB)) // if fieldA > fieldB then return the asine of fieldA, else return the asine of fieldB
-----
-
-== atan
-
-The `atan` function returns the trigonometric arctangent of a number.
-
-=== atan Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the arctangent of.
-
-=== atan Syntax
-
-[source,text]
-----
-atan(100.4) // returns the arctangent of 100.4
-atan(fieldA) // returns the arctangent for fieldA.
-if(gt(fieldA,fieldB),atan(fieldA),atan(fieldB)) // if fieldA > fieldB then return the arctanget of fieldA, else return the arctangent of fieldB
-----
-
-== cbrt
-
-The `cbrt` function returns the trigonometric cube root of a number.
-
-=== cbrt Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the cube root of.
-
-=== cbrt Syntax
-
-[source,text]
-----
-cbrt(100.4) // returns the square root of 100.4
-cbrt(fieldA) // returns the square root for fieldA.
-if(gt(fieldA,fieldB),cbrt(fieldA),cbrt(fieldB)) // if fieldA > fieldB then return the cbrt of fieldA, else return the cbrt of fieldB
-----
-
-== ceil
-The `ceil` function rounds a decimal value to the next highest whole number.
-
-=== ceil Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The decimal to round up.
-
-=== ceil Syntax
-
-The expressions below show the various ways in which you can use the `ceil` evaluator.
-
-[source,text]
-----
-ceil(100.4) // returns 101.
-ceil(fieldA) // returns the next highest whole number for fieldA.
-if(gt(fieldA,fieldB),ceil(fieldA),ceil(fieldB)) // if fieldA > fieldB then return the ceil of fieldA, else return the ceil of fieldB.
-----
-
-== col
-
-The `col` function returns a numeric array from a list of Tuples. The `col`
-function is used to create numeric arrays from stream sources.
-
-=== col Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `list of Tuples`
-* `field name`: The field to create the array from.
-
-
-=== col Syntax
-
-[source,text]
-col(tupleList, fieldName)
-
-== conv
-
-The `conv` function returns the https://en.wikipedia.org/wiki/Convolution[convolution] of two numeric arrays.
-
-=== conv Parameters
-
-* `numeric array`
-* `numeric array`
-
-=== conv Syntax
-
-[source,text]
-conv(numericArray1, numericArray2)
-
-== copyOf
-
-The `copyOf` function creates a copy of a numeric array.
-
-=== copyOf Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-* `length`: The length of the copied array. The returned array will be right padded with zeros if the length parameter exceeds the size of the original array.
-
-=== copyOf Syntax
-
-[source,text]
-copyOf(numericArray, length)
-
-== copyOfRange
-
-The `copyOfRange` function creates a copy of a range of a numeric array.
-
-=== copyOfRange Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-* `start index`
-* `end index`
-
-=== copyOfRange Syntax
-
-[source,text]
-copyOfRange(numericArray, startIndex, endIndex)
-
-== corr
-
-The `corr` function returns the Pearson Product Moment Correlation of two numeric arrays.
-
-=== corr Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-* `numeric array`
-
-=== corr Returns
-
-double between -1 and 1
-
-=== corr Synax
-
-[source,text]
-corr(numericArray1, numericArray2)
-
-== cos
-The `cos` function returns the trigonometric cosine of a number.
-
-=== cos Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the hyperbolic cosine of.
-
-=== cos Syntax
-
-[source,text]
-----
-cos(100.4) // returns the arccosine of 100.4
-cos(fieldA) // returns the arccosine for fieldA.
-if(gt(fieldA,fieldB),cos(fieldA),cos(fieldB)) // if fieldA > fieldB then return the arccosine of fieldA, else return the cosine of fieldB
-----
-
-== cov
-
-The `cov` function returns the covariance of two numeric arrays.
-
-=== cov Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-* `numeric array`
-
-=== cov Syntax
-
-[source,text]
-cov(numericArray, numericArray)
-
-== describe
-
-The `describe` function returns a tuple containing the descriptive statistics for an array.
-
-=== describe Parameters
-
-* `numeric array`
-
-=== describe Syntax
-
-[source,text]
-describe(numericArray)
-
-== distance
-
-The `distance` function calculates the Euclidian distance of two numeric arrays.
-
-=== distance Parameters
-
-* `numeric array`
-* `numeric array`
-
-=== distance Syntax
-
-[source,text]
-distance(numericArray1, numuericArray2))
-
-== div
-
-The `div` function will take two numeric values and divide them. The function will fail to execute if any of the values are non-numeric or null, or the 2nd value is 0. Returns a numeric value.
-
-=== div Parameters
-
-* `Field Name | Raw Number | Number Evaluator`
-* `Field Name | Raw Number | Number Evaluator`
-
-=== div Syntax
-
-The expressions below show the various ways in which you can use the `div` evaluator. The first value will be divided by the second and as such the second cannot be 0.
-
-[source,text]
-----
-div(1,2) // 1 / 2
-div(1,fieldA) // 1 / fieldA
-div(fieldA,1.4) // fieldA / 1.4
-div(fieldA,add(fieldA,fieldB)) // fieldA / (fieldA + fieldB)
-----
-
-== eor
-
-The `eor` function will return the logical exclusive or of at least two boolean parameters. The function will fail to execute if any parameters are non-boolean or null. Returns a boolean value.
-
-=== eor Parameters
-
-* `Field Name | Raw Boolean | Boolean Evaluator`
-* `Field Name | Raw Boolean | Boolean Evaluator`
-* `......`
-* `Field Name | Raw Boolean | Boolean Evaluator`
-
-=== eor Syntax
-
-The expressions below show the various ways in which you can use the `eor` evaluator. At least two parameters are required, but there is no limit to how many you can use.
-
-[source,text]
-----
-eor(true,fieldA) // true iff fieldA is false
-eor(fieldA,fieldB) // true iff either fieldA or fieldB is true but not both
-eor(eq(fieldA,fieldB),eq(fieldC,fieldD)) // true iff either fieldA == fieldB or fieldC == fieldD but not both
-----
-
-== eq
-
-The `eq` function will return whether all the parameters are equal, as per Java's standard `equals(...)` function. The function accepts parameters of any type, but will fail to execute if all the parameters are not of the same type. That is, all are Boolean, all are String, all are Numeric. If any any parameters are null and there is at least one parameter that is not null then false will be returned. Returns a boolean value.
-
-=== eq Parameters
-
-* `Field Name | Raw Value | Evaluator`
-* `Field Name | Raw Value | Evaluator`
-* `......`
-* `Field Name | Raw Value | Evaluator`
-
-=== eq Syntax
-
-The expressions below show the various ways in which you can use the `eq` evaluator.
-
-[source,text]
-----
-eq(1,2) // 1 == 2
-eq(1,fieldA) // 1 == fieldA
-eq(fieldA,val(foo)) fieldA == "foo"
-eq(add(fieldA,fieldB),6) // fieldA + fieldB == 6
-----
-
-== finddelay
-
-The `finddelay` function performs a cross-correlation between two numeric arrays and returns the delay.
-
-=== finddelay Parameters
-
-* `numeric array`
-* `numeric array`
-
-=== finddelay Syntax
-
-[source,text]
-finddelay(numericArray1, numericArray2)
-
-== floor
-The `floor` function rounds a decimal value to the next lowest whole number.
-
-=== floor Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The decimal to round down.
-
-=== floor Syntax
-
-The expressions below show the various ways in which you can use the `floor` evaluator.
-
-[source,text]
-----
-floor(100.4) // returns 100.
-ceil(fieldA) // returns the next lowestt whole number for fieldA.
-if(gt(fieldA,fieldB),floor(fieldA),floor(fieldB)) // if fieldA > fieldB then return the floor of fieldA, else return the floor of fieldB.
-----
-
-== gt
-
-The `gt` function will return whether the first parameter is greater than the second parameter. The function accepts numeric or string parameters, but will fail to execute if all the parameters are not of the same type. That is, all are String or all are Numeric. If any any parameters are null then an error will be raised. Returns a boolean value.
-
-=== gt Parameters
-
-* `Field Name | Raw Value | Evaluator`
-* `Field Name | Raw Value | Evaluator`
-
-=== gt Syntax
-
-The expressions below show the various ways in which you can use the `gt` evaluator.
-
-[source,text]
-----
-gt(1,2) // 1 > 2
-gt(1,fieldA) // 1 > fieldA
-gt(fieldA,val(foo)) // fieldA > "foo"
-gt(add(fieldA,fieldB),6) // fieldA + fieldB > 6
-----
-
-== gteq
-
-The `gteq` function will return whether the first parameter is greater than or equal to the second parameter. The function accepts numeric and string parameters, but will fail to execute if all the parameters are not of the same type. That is, all are String or all are Numeric. If any any parameters are null then an error will be raised. Returns a boolean value.
-
-=== gteq Parameters
-
-* `Field Name | Raw Value | Evaluator`
-* `Field Name | Raw Value | Evaluator`
-
-=== gteq Syntax
-
-The expressions below show the various ways in which you can use the `gteq` evaluator.
-
-[source,text]
-----
-gteq(1,2) // 1 >= 2
-gteq(1,fieldA) // 1 >= fieldA
-gteq(fieldA,val(foo)) fieldA >= "foo"
-gteq(add(fieldA,fieldB),6) // fieldA + fieldB >= 6
-----
-
-== hist
-
-The `hist` function creates a histogram from a numeric array. The hist function is designed
-to work with continuous variables.
-
-=== hist Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-* `bins`: The number of bins in the histogram. Each returned tuple contains
-summary statistics for the observations that were within the bin.
-
-=== hist Sytnax
-
-[source,text]
-hist(numericArray, bins)
-
-== hsin
-The `hsin` function returns the trigonometric hyperbolic sine of a number.
-
-=== hsin Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the hyperbolic sine of.
-
-=== hsin Syntax
-
-[source,text]
-----
-hsin(100.4) // returns the hsine of 100.4
-hsin(fieldA) // returns the hsine for fieldA.
-if(gt(fieldA,fieldB),sin(fieldA),sin(fieldB)) // if fieldA > fieldB then return the hsine of fieldA, else return the hsine of fieldB
-----
-
-== if
-
-The `if` function works like a standard conditional if/then statement. If the first parameter is true, then the second parameter will be returned, else the third parameter will be returned. The function accepts a boolean as the first parameter and anything as the second and third parameters. An error will occur if the first parameter is not a boolean or is null.
-
-=== if Parameters
-
-* `Field Name | Raw Value | Boolean Evaluator`
-* `Field Name | Raw Value | Evaluator`
-* `Field Name | Raw Value | Evaluator`
-
-=== if Syntax
-
-The expressions below show the various ways in which you can use the `if` evaluator.
-
-[source,text]
-----
-if(fieldA,fieldB,fieldC) // if fieldA is true then fieldB else fieldC
-if(gt(fieldA,5), fieldA, 5) // if fieldA > 5 then fieldA else 5
-if(eq(fieldB,null), null, div(fieldA,fieldB)) // if fieldB is null then null else fieldA / fieldB
-----
-
-== length
-
-The `length` function returns the length of a numeric array.
-
-=== length Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-
-=== length Syntax
-
-[source,text]
-length(numericArray)
-
-== log
-
-The `log` function will return the natural log of the provided single parameter. The `log` function will fail to execute if the value is non-numeric. If a null value is found, then null will be returned as the result.
-
-=== log Parameters
-
-* `Field Name | Raw Number | Number Evaluator`
-
-=== log Syntax
-
-The expressions below show the various ways in which you can use the `log` evaluator. Only one parameter is accepted. Returns a numeric value.
-
-[source,text]
-----
-log(100)
-log(add(fieldA,fieldB))
-log(fieldA)
-----
-
-== lt
-
-The `lt` function will return whether the first parameter is less than the second parameter. The function accepts numeric or string parameters, but will fail to execute if all the parameters are not of the same type. That is, all are String or all are Numeric. If any any parameters are null then an error will be raised. Returns a boolean value.
-
-=== lt Parameters
-
-* `Field Name | Raw Value | Evaluator`
-* `Field Name | Raw Value | Evaluator`
-
-=== lt Syntax
-
-The expressions below show the various ways in which you can use the `lt` evaluator.
-
-[source,text]
-----
-lt(1,2) // 1 < 2
-lt(1,fieldA) // 1 < fieldA
-lt(fieldA,val(foo)) fieldA < "foo"
-lt(add(fieldA,fieldB),6) // fieldA + fieldB < 6
-----
-
-== lteq
-
-The `lteq` function will return whether the first parameter is less than or equal to the second parameter. The function accepts numeric and string parameters, but will fail to execute if all the parameters are not of the same type. That is, all are String or all are Numeric. If any any parameters are null then an error will be raised. Returns a boolean value.
-
-=== lteq Parameters
-
-* `Field Name | Raw Value | Evaluator`
-* `Field Name | Raw Value | Evaluator`
-
-=== lteq Syntax
-
-The expressions below show the various ways in which you can use the `lteq` evaluator.
-
-[source,text]
-----
-lteq(1,2) // 1 <= 2
-lteq(1,fieldA) // 1 <= fieldA
-lteq(fieldA,val(foo)) fieldA <= "foo"
-lteq(add(fieldA,fieldB),6) // fieldA + fieldB <= 6
-----
-
-== mod
-The `mod` function returns the remainder (modulo) of the first parameter divided by the second parameter.
-
-=== mod Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: Parameter 1
-* `Field Name | Raw Number | Number Evaluator`: Parameter 2
-
-=== mod Syntax
-
-The expressions below show the various ways in which you can use the `mod` evaluator.
-
-[source,text]
-----
-mod(100,3) // returns the remainder of 100 / 3 .
-mod(100,fieldA) // returns the remainder of 100 divided by the value of fieldA.
-mod(fieldA,1.4) // returns the remainder of fieldA divided by 1.4.
-if(gt(fieldA,fieldB),mod(fieldA,fieldB),mod(fieldB,fieldA)) // if fieldA > fieldB then return the remainder of fieldA/fieldB, else return the remainder of fieldB/fieldA.
-----
-
-== movingAvg
-
-The `movingAvg` function calculates a https://en.wikipedia.org/wiki/Moving_average[moving average] over an array of numbers.
-
-=== movingAvg Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-* `window size`: The array returned will be smaller than this value.
-
-=== movingAvg Syntax
-
-[source,text]
-movingAverage(numericArray, 30)
-
-== mult
-
-The `mult` function will take two or more numeric values and multiply them together. The `mult` function will fail to execute if any of the values are non-numeric. If a null value is found then null will be returned as the result.
-
-=== mult Parameters
-
-* `Field Name | Raw Number | Number Evaluator`
-* `Field Name | Raw Number | Number Evaluator`
-* `......`
-* `Field Name | Raw Number | Number Evaluator`
-
-=== mult Syntax
-
-The expressions below show the various ways in which you can use the `mult` evaluator. The number and order of these parameters do not matter and is not limited except that at least two parameters are required. Returns a numeric value.
-
-[source,text]
-----
-mult(1,2,3,4) // 1 * 2 * 3 * 4
-mult(1,fieldA) // 1 * value of fieldA
-mult(fieldA,1.4) // value of fieldA * 1.4
-mult(fieldA,fieldB,fieldC) // value of fieldA * value of fieldB * value of fieldC
-mult(fieldA,div(fieldA,fieldB)) // value of fieldA * (value of fieldA / value of fieldB)
-mult(fieldA,if(gt(fieldA,fieldB),fieldA,fieldB)) // if fieldA > fieldB then fieldA * fieldA, else fieldA * fieldB
-----
-
-== normalize
-
-The `normalize` function normalizes a numeric array so that values within the array
-have a mean of 0 and standard deviation of 1.
-
-=== normalize Parameters
-
-* `numeric array`
-
-=== normalize Syntax
-
-[source,text]
-normalize(numericArray)
-
-== not
-
-The `not` function will return the logical NOT of a single boolean parameter. The function will fail to execute if the parameter is non-boolean or null. Returns a boolean value.
-
-=== not Parameters
-
-* `Field Name | Raw Boolean | Boolean Evaluator`
-
-=== not Syntax
-
-The expressions below show the various ways in which you can use the `not` evaluator. Only one parameter is allowed.
-
-[source,text]
-----
-not(true) // false
-not(fieldA) // true if fieldA is false else false
-not(eq(fieldA,fieldB)) // true if fieldA != fieldB
-----
-
-== or
-
-The `or` function will return the logical OR of at least 2 boolean parameters. The function will fail to execute if any parameters are non-boolean or null. Returns a boolean value.
-
-=== or Parameters
-
-* `Field Name | Raw Boolean | Boolean Evaluator`
-* `Field Name | Raw Boolean | Boolean Evaluator`
-* `......`
-* `Field Name | Raw Boolean | Boolean Evaluator`
-
-=== or Syntax
-
-The expressions below show the various ways in which you can use the `or` evaluator. At least two parameters are required, but there is no limit to how many you can use.
-
-[source,text]
-----
-or(true,fieldA) // true || fieldA
-or(fieldA,fieldB) // fieldA || fieldB
-or(and(fieldA,fieldB),fieldC) // (fieldA && fieldB) || fieldC
-or(fieldA,fieldB,fieldC,and(fieldD,fieldE),fieldF)
-----
-
-== predict
-
-The `predict` function predicts the value of an dependent variable based on
-the output of the regress function.
-
-=== predict Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `regress output`
-* `numeric predictor`
-
-=== predict Syntax
-
-[source,text]
-predict(regressOutput, predictor)
-
-== pow
-The `pow` function returns the value of its first parameter raised to the power of its second parameter.
-
-=== pow Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: Parameter 1
-* `Field Name | Raw Number | Number Evaluator`: Parameter 2
-
-=== pow Syntax
-
-The expressions below show the various ways in which you can use the `pow` evaluator.
-
-[source,text]
-----
-pow(2,3) // returns 2 raised to the 3rd power.
-pow(4,fieldA) // returns 4 raised by the value of fieldA.
-pow(fieldA,1.4) // returns the value of fieldA raised by 1.4.
-if(gt(fieldA,fieldB),pow(fieldA,fieldB),pow(fieldB,fieldA)) // if fieldA > fieldB then raise fieldA by fieldB, else raise fieldB by fieldA.
-----
-
-== rank
-
-The `rank` performs a rank transformation on a numeric array.
-
-=== rank Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-
-=== rank Syntax
-
-[source,text]
-rank(numericArray)
-
-== raw
-
-The `raw` function will return whatever raw value is the parameter. This is useful for cases where you want to use a string as part of another evaluator.
-
-=== raw Parameters
-
-* `Raw Value`
-
-=== raw Syntax
-
-The expressions below show the various ways in which you can use the `raw` evaluator. Whatever is inside will be returned as-is. Internal evaluators are considered strings and are not evaluated.
-
-[source,text]
-----
-raw(foo) // "foo"
-raw(count(*)) // "count(*)"
-raw(45) // 45
-raw(true) // "true" (note: this returns the string "true" and not the boolean true)
-eq(raw(fieldA), fieldA) // true if the value of fieldA equals the string "fieldA"
-----
-
-== regress
-
-The `regress` function performs a simple regression on two numeric arrays.
-
-The result of this expression is also used by the `predict` function.
-
-=== regress Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-* `numeric array`
-
-=== regress Syntax
-
-[source,text]
-regress(numericArray1, numericArray2)
-
-== rev
-
-The `rev` function reverses the order of a numeric array.
-
-=== rev Parameters
-
-* `numeric array`
-
-=== rev Syntax
-
-[source,text]
-rev(numericArray)
-
-== round
-
-The `round` function returns the closest whole number to the argument
-
-=== round Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the square root of.
-
-=== round Syntax
-
-[source,text]
-----
-round(100.4)
-round(fieldA)
-if(gt(fieldA,fieldB),sqrt(fieldA),sqrt(fieldB)) // if fieldA > fieldB then return the round of fieldA, else return the round of fieldB
-----
-
-== scale
-
-The `scale` function multiplies all the elements of an array by a number.
-
-=== scale Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `number`
-* `numeric array`
-
-=== scale Syntax
-
-[source,text]
-scale(number, numericArray)
-
-== sequence
-
-The `sequence` function returns an array of numbers based on its parameters.
-
-=== sequence Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `length`
-* `start`
-* `stride`
-
-=== sequence Syntax
-
-[source,text]
-sequence(100, 0, 1) // Returns a sequence of length 100, starting from 0 with a stride of 1.
-
-== sin
-The `sin` function returns the trigonometric sine of a number.
-
-=== sin Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the sine of.
-
-=== sin Syntax
-
-[source,text]
-----
-sin(100.4) // returns the sine of 100.4
-sine(fieldA) // returns the sine for fieldA.
-if(gt(fieldA,fieldB),sin(fieldA),sin(fieldB)) // if fieldA > fieldB then return the sine of fieldA, else return the sine of fieldB
-----
-
-== sqrt
-
-The `sqrt` function returns the trigonometric square root of a number.
-
-=== sqrt Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the square root of.
-
-=== sqrt Syntax
-
-[source,text]
-----
-sqrt(100.4) // returns the square root of 100.4
-sqrt(fieldA) // returns the square root for fieldA.
-if(gt(fieldA,fieldB),sqrt(fieldA),sqrt(fieldB)) // if fieldA > fieldB then return the sqrt of fieldA, else return the sqrt of fieldB
-----
-
-== sub
-
-The `sub` function will take 2 or more numeric values and subtract them, from left to right. The sub function will fail to execute if any of the values are non-numeric. If a null value is found then null will be returned as the result.
-
-=== sub Parameters
-
-* `Field Name | Raw Number | Number Evaluator`
-* `Field Name | Raw Number | Number Evaluator`
-* `......`
-* `Field Name | Raw Number | Number Evaluator`
-
-=== sub Syntax
-
-The expressions below show the various ways in which you can use the `sub` evaluator. The number of these parameters does not matter and is not limited except that at least two parameters are required. Returns a numeric value.
-
-[source,text]
-----
-sub(1,2,3,4) // 1 - 2 - 3 - 4
-sub(1,fieldA) // 1 - value of fieldA
-sub(fieldA,1.4) // value of fieldA - 1.4
-sub(fieldA,fieldB,fieldC) // value of fieldA - value of fieldB - value of fieldC
-sub(fieldA,div(fieldA,fieldB)) // value of fieldA - (value of fieldA / value of fieldB)
-if(gt(fieldA,fieldB),sub(fieldA,fieldB),sub(fieldB,fieldA)) // if fieldA > fieldB then fieldA - fieldB, else fieldB - field
-----
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9beafb61/solr/solr-ref-guide/src/stream-source-reference.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/stream-source-reference.adoc b/solr/solr-ref-guide/src/stream-source-reference.adoc
new file mode 100644
index 0000000..cf6e451
--- /dev/null
+++ b/solr/solr-ref-guide/src/stream-source-reference.adoc
@@ -0,0 +1,539 @@
+= Stream Source Reference
+:page-shortname: stream-source-reference
+:page-permalink: stream-source-reference.html
+:page-tocclass: right
+:page-toclevels: 1
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+Put something here to see if it fixes things.
+
+== search
+
+The `search` function searches a SolrCloud collection and emits a stream of tuples that match the query. This is very similar to a standard Solr query, and uses many of the same parameters.
+
+This expression allows you to specify a request hander using the `qt` parameter. By default, the `/select` handler is used. The `/select` handler can be used for simple rapid prototyping of expressions. For production, however, you will most likely want to use the `/export` handler which is designed to `sort` and `export` entire result sets. The `/export` handler is not used by default because it has stricter requirements then the `/select` handler so it's not as easy to get started working with. To read more about the `/export` handler requirements review the section <<exporting-result-sets.adoc#exporting-result-sets,Exporting Result Sets>>.
+
+=== search Parameters
+
+* `collection`: (Mandatory) the collection being searched.
+* `q`: (Mandatory) The query to perform on the Solr index.
+* `fl`: (Mandatory) The list of fields to return.
+* `sort`: (Mandatory) The sort criteria.
+* `zkHost`: Only needs to be defined if the collection being searched is found in a different zkHost than the local stream handler.
+* `qt`: Specifies the query type, or request handler, to use. Set this to `/export` to work with large result sets. The default is `/select`.
+* `rows`: (Mandatory with the `/select` handler) The rows parameter specifies how many rows to return. This parameter is only needed with the `/select` handler (which is the default) since the `/export` handler always returns all rows.
+* `partitionKeys`: Comma delimited list of keys to partition the search results by. To be used with the parallel function for parallelizing operations across worker nodes. See the <<stream-decorator-reference.adoc#parallel,parallel>> function for details.
+
+=== search Syntax
+
+[source,text]
+----
+expr=search(collection1,
+ zkHost="localhost:9983",
+ qt="/export",
+ q="*:*",
+ fl="id,a_s,a_i,a_f",
+ sort="a_f asc, a_i asc")
+----
+
+== jdbc
+
+The `jdbc` function searches a JDBC datasource and emits a stream of tuples representing the JDBC result set. Each row in the result set is translated into a tuple and each tuple contains all the cell values for that row.
+
+=== jdbc Parameters
+
+* `connection`: (Mandatory) JDBC formatted connection string to whatever driver you are using.
+* `sql`: (Mandatory) query to pass off to the JDBC endpoint
+* `sort`: (Mandatory) The sort criteria indicating how the data coming out of the JDBC stream is sorted
+* `driver`: The name of the JDBC driver used for the connection. If provided then the driver class will attempt to be loaded into the JVM. If not provided then it is assumed that the driver is already loaded into the JVM. Some drivers require explicit loading so this option is provided.
+* `[driverProperty]`: One or more properties to pass to the JDBC driver during connection. The format is `propertyName="propertyValue"`. You can provide as many of these properties as you'd like and they will all be passed to the connection.
+
+=== Connections and Drivers
+
+Because some JDBC drivers require explicit loading the `driver` parameter can be used to provide the driver class name. If provided, then during stream construction the driver will be loaded. If the driver cannot be loaded because the class is not found on the classpath, then stream construction will fail.
+
+When the JDBC stream is opened it will validate that a driver can be found for the provided connection string. If a driver cannot be found (because it hasn't been loaded) then the open will fail.
+
+=== Datatypes
+
+Due to the inherent differences in datatypes across JDBC sources the following datatypes are supported. The table indicates what Java type will be used for a given JDBC type. Types marked as requiring conversion will go through a conversion for each value of that type. For performance reasons the cell data types are only considered when the stream is opened as this is when the converters are created.
+
+[width="100%",options="header",]
+|===
+|JDBC Type |Java Type |Requires Conversion
+|String |String |No
+|Short |Long |Yes
+|Integer |Long |Yes
+|Long |Long |No
+|Float |Double |Yes
+|Double |Double |No
+|Boolean |Boolean |No
+|===
+
+=== jdbc Syntax
+
+A basic `jdbc` expression:
+
+[source,text]
+----
+jdbc(
+ connection="jdbc:hsqldb:mem:.",
+ sql="select NAME, ADDRESS, EMAIL, AGE from PEOPLE where AGE > 25 order by AGE, NAME DESC",
+ sort="AGE asc, NAME desc",
+ driver="org.hsqldb.jdbcDriver"
+)
+----
+
+A `jdbc` expression that passes a property to the driver:
+
+[source,text]
+----
+// get_column_name is a property to pass to the hsqldb driver
+jdbc(
+ connection="jdbc:hsqldb:mem:.",
+ sql="select NAME as FIRST_NAME, ADDRESS, EMAIL, AGE from PEOPLE where AGE > 25 order by AGE, NAME DESC",
+ sort="AGE asc, NAME desc",
+ driver="org.hsqldb.jdbcDriver",
+ get_column_name="false"
+)
+----
+
+== echo
+
+The `echo` function returns a single Tuple echoing its text parameter. `Echo` is the simplest stream source designed to provide text
+to a text analyzing stream decorator.
+
+=== echo Syntax
+
+[source,text]
+----
+echo("Hello world")
+----
+
+== facet
+
+The `facet` function provides aggregations that are rolled up over buckets. Under the covers the facet function pushes down the aggregation into the search engine using Solr's JSON Facet API. This provides sub-second performance for many use cases. The facet function is appropriate for use with a low to moderate number of distinct values in the bucket fields. To support high cardinality aggregations see the rollup function.
+
+=== facet Parameters
+
+* `collection`: (Mandatory) Collection the facets will be aggregated from.
+* `q`: (Mandatory) The query to build the aggregations from.
+* `buckets`: (Mandatory) Comma separated list of fields to rollup over. The comma separated list represents the dimensions in a multi-dimensional rollup.
+* `bucketSorts`: Comma separated list of sorts to apply to each dimension in the buckets parameters. Sorts can be on the computed metrics or on the bucket values.
+* `bucketSizeLimit`: The number of buckets to include. This value is applied to each dimension.
+* `metrics`: List of metrics to compute for the buckets. Currently supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)`, `count(*)`.
+
+=== facet Syntax
+
+Example 1:
+
+[source,text]
+----
+facet(collection1,
+ q="*:*",
+ buckets="a_s",
+ bucketSorts="sum(a_i) desc",
+ bucketSizeLimit=100,
+ sum(a_i),
+ sum(a_f),
+ min(a_i),
+ min(a_f),
+ max(a_i),
+ max(a_f),
+ avg(a_i),
+ avg(a_f),
+ count(*))
+----
+
+The example above shows a facet function with rollups over a single bucket, where the buckets are returned in descending order by the calculated value of the `sum(a_i)` metric.
+
+Example 2:
+
+[source,text]
+----
+facet(collection1,
+ q="*:*",
+ buckets="year_i, month_i, day_i",
+ bucketSorts="year_i desc, month_i desc, day_i desc",
+ bucketSizeLimit=100,
+ sum(a_i),
+ sum(a_f),
+ min(a_i),
+ min(a_f),
+ max(a_i),
+ max(a_f),
+ avg(a_i),
+ avg(a_f),
+ count(*))
+----
+
+The example above shows a facet function with rollups over three buckets, where the buckets are returned in descending order by bucket value.
+
+== features
+
+The `features` function extracts the key terms from a text field in a classification training set stored in a SolrCloud collection. It uses an algorithm known as * Information Gain* , to select the important terms from the training set. The `features` function was designed to work specifically with the <<train,train>> function, which uses the extracted features to train a text classifier.
+
+The `features` function is designed to work with a training set that provides both positive and negative examples of a class. It emits a tuple for each feature term that is extracted along with the inverse document frequency (IDF) for the term in the training set.
+
+The `features` function uses a query to select the training set from a collection. The IDF for each selected feature is calculated relative to the training set matching the query. This allows multiple training sets to be stored in the same SolrCloud collection without polluting the IDF across training sets.
+
+=== features Parameters
+
+* `collection`: (Mandatory) The collection that holds the training set
+* `q`: (Mandatory) The query that defines the training set. The IDF for the features will be generated specific to the result set matching the query.
+* `featureSet`: (Mandatory) The name of the feature set. This can be used to retrieve the features if they are stored in a SolrCloud collection.
+* `field`: (Mandatory) The text field to extract the features from.
+* `outcome`: (Mandatory) The field that defines the class, positive or negative
+* `numTerms`: (Mandatory) How many feature terms to extract.
+* `positiveLabel`: (defaults to 1) The value in the outcome field that defines a postive outcome.
+
+=== features Syntax
+
+[source,text]
+----
+features(collection1,
+ q="*:*",
+ featureSet="features1",
+ field="body",
+ outcome="out_i",
+ numTerms=250)
+----
+
+== nodes
+
+The `nodes` function provides breadth-first graph traversal. For details, see the section <<graph-traversal.adoc#graph-traversal,Graph Traversal>>.
+
+== knn
+
+The `knn` function returns the K nearest neighbors for a document based on text similarity. Under the covers the `knn` function
+use the More Like This query parser plugin.
+
+=== knn Parameters
+
+* `collection`: (Mandatory) The collection to perform the search in.
+* `id`: (Mandatory) The id of the source document to begin the knn search from.
+* `qf`: (Mandatory) The query field used to compare documents.
+* `k`: (Mandatory) The number of nearest neighbors to return.
+* `fl`: (Mandatory) The field list to return.
+* `mintf`: (Optional) The minimum numer of occurrences of the term in the source document to be inlcued in the search.
+* `maxtf`: (Optional) The maximum numer of occurrences of the term in the source document to be inlcued in the search.
+* `mindf`: (Optional) The minimum numer of occurrences in the corpus to be inlcued in the search.
+* `maxdf`: (Optional) The maximum numer of occurrences in the corpus to be inlcued in the search.
+* `minwl`: (Optional) The minimum world length of to be inlcued in the search.
+* `maxwl`: (Optional) The maximum world length of to be inlcued in the search.
+
+=== knn Syntax
+
+[source,text]
+----
+knn(collection1,
+ id="doc1",
+ qf="text_field",
+ k="10",
+ fl="id, title",
+ mintf="3",
+ maxdf="10000000")
+----
+
+== model
+
+The `model` function retrieves and caches logistic regression text classification models that are stored in a SolrCloud collection. The `model` function is designed to work with models that are created by the <<train,train function>>, but can also be used to retrieve text classification models trained outside of Solr, as long as they conform to the specified format. After the model is retrieved it can be used by the <<stream-decorator-reference.adoc#classify,classify function>> to classify documents.
+
+A single model tuple is fetched and returned based on the *id* parameter. The model is retrieved by matching the *id* parameter with a model name in the index. If more then one iteration of the named model is stored in the index, the highest iteration is selected.
+
+=== Caching with model
+
+The `model` function has an internal LRU (least-recently-used) cache so models do not have to be retrieved with each invocation of the `model` function. The time to cache for each model ID can be passed as a parameter to the function call. Retrieving a cached model does not reset the time for expiring the model ID in the cache.
+
+=== Model Storage
+
+The storage format of the models in Solr is below. The `train` function outputs the format below so you only need to know schema details if you plan to use the `model` function with logistic regression models trained outside of Solr.
+
+* `name_s` (Single value, String, Stored): The name of the model.
+* `iteration_i` (Single value, Integer, Stored): The iteration number of the model. Solr can store all iterations of the models generated by the train function.
+* `terms_ss` (Multi value, String, Stored: The array of terms/features of the model.
+* `weights_ds` (Multi value, double, Stored): The array of term weights. Each weight corresponds by array index to a term.
+* `idfs_ds` (Multi value, double, Stored): The array of term IDFs (Inverse document frequency). Each IDF corresponds by array index to a term.
+
+=== model Parameters
+
+* `collection`: (Mandatory) The collection where the model is stored.
+* `id`: (Mandatory) The id/name of the model. The model function always returns one model. If there are multiple iterations of the name, the highest iteration is returned.
+* `cacheMillis`: (Optional) The amount of time to cache the model in the LRU cache.
+
+=== model Syntax
+
+[source,text]
+----
+model(modelCollection,
+ id="myModel"
+ cacheMillis="200000")
+----
+
+== random
+
+The `random` function searches a SolrCloud collection and emits a pseudo-random set of results that match the query. Each invocation of random will return a different pseudo-random result set.
+
+=== random Parameters
+
+* `collection`: (Mandatory) The collection the stats will be aggregated from.
+* `q`: (Mandatory) The query to build the aggregations from.
+* `rows`: (Mandatory) The number of pseudo-random results to return.
+* fl: (Mandatory) The field list to return.
+* `fq`: (Optional) Filter query
+
+=== random Syntax
+
+[source,text]
+----
+random(baskets,
+ q="productID:productX",
+ rows="100",
+ fl="basketID")
+----
+
+In the example above the `random` function is searching the baskets collections for all rows where "productID:productX". It will return 100 pseudo-random results. The field list returned is the basketID.
+
+== significantTerms
+
+The `significantTerms` function queries a SolrCloud collection, but instead of returning documents, it returns significant terms found in documents in the result set. The `significantTerms` function scores terms based on how frequently they appear in the result set and how rarely they appear in the entire corpus. The `significantTerms` function emits a tuple for each term which contains the term, the score, the foreground count and the background count. The foreground count is how many documents the term appears in in the result set. The background count is how many documents the term appears in in the entire corpus. The foreground and background counts are global for the collection.
+
+=== significantTerms Parameters
+
+* `collection`: (Mandatory) The collection that the function is run on.
+* `q`: (Mandatory) The query that describes the foreground document set.
+* `field`: (Mandatory) The field to extract the terms from.
+* `limit`: (Optional, Default 20) The max number of terms to return.
+* `minDocFreq`: (Optional, Defaults to 5 documents) The minimum number of documents the term must appear in on a shard. This is a float value. If greater then 1.0 then it's considered the absolute number of documents. If less then 1.0 it's treated as a percentage of documents.
+* `maxDocFreq`: (Optional, Defaults to 30% of documents) The maximum number of documents the term can appear in on a shard. This is a float value. If greater then 1.0 then it's considered the absolute number of documents. If less then 1.0 it's treated as a percentage of documents.
+* `minTermLength`: (Optional, Default 4) The minimum length of the term to be considered significant.
+
+=== significantTerms Syntax
+
+[source,text]
+----
+significantTerms(collection1,
+ q="body:Solr",
+ field="author",
+ limit="50",
+ minDocFreq="10",
+ maxDocFreq=".20",
+ minTermLength="5")
+----
+
+In the example above the `significantTerms` function is querying `collection1` and returning at most 50 significant terms from the `authors` field that appear in 10 or more documents but not more then 20% of the corpus.
+
+== shortestPath
+
+The `shortestPath` function is an implementation of a shortest path graph traversal. The `shortestPath` function performs an iterative breadth-first search through an unweighted graph to find the shortest paths between two nodes in a graph. The `shortestPath` function emits a tuple for each path found. Each tuple emitted will contain a `path` key which points to a `List` of nodeIDs comprising the path.
+
+=== shortestPath Parameters
+
+* `collection`: (Mandatory) The collection that the topic query will be run on.
+* `from`: (Mandatory) The nodeID to start the search from
+* `to`: (Mandatory) The nodeID to end the search at
+* `edge`: (Mandatory) Syntax: `from_field=to_field`. The `from_field` defines which field to search from. The `to_field` defines which field to search to. See example below for a detailed explanation.
+* `threads`: (Optional : Default 6) The number of threads used to perform the partitioned join in the traversal.
+* `partitionSize`: (Optional : Default 250) The number of nodes in each partition of the join.
+* `fq`: (Optional) Filter query
+* `maxDepth`: (Mandatory) Limits to the search to a maximum depth in the graph.
+
+=== shortestPath Syntax
+
+[source,text]
+----
+shortestPath(collection,
+ from="john@company.com",
+ to="jane@company.com",
+ edge="from_address=to_address",
+ threads="6",
+ partitionSize="300",
+ fq="limiting query",
+ maxDepth="4")
+----
+
+The expression above performs a breadth-first search to find the shortest paths in an unweighted, directed graph.
+
+The search starts from the nodeID "\john@company.com" in the `from_address` field and searches for the nodeID "\jane@company.com" in the `to_address` field. This search is performed iteratively until the `maxDepth` has been reached. Each level in the traversal is implemented as a parallel partitioned nested loop join across the entire collection. The `threads` parameter controls the number of threads performing the join at each level, while the `partitionSize` parameter controls the of number of nodes in each join partition. The `maxDepth` parameter controls the number of levels to traverse. `fq` is a limiting query applied to each level in the traversal.
+
+== shuffle
+
+The `shuffle` expression sorts and exports entire result sets. The `shuffle` expression is similar to the `search` expression except that
+under the covers `shuffle` always uses the /export handler. The `shuffle` expression is designed to be combined with the relational algebra
+decorators that require complete, sorted result sets. Shuffled result sets can be partitioned across worker nodes with the parallel
+stream decorator to perform parallel relational algebra. When used in parallel mode the partitionKeys parameter must be provided.
+
+=== shuffle Parameters
+
+* `collection`: (Mandatory) the collection being searched.
+* `q`: (Mandatory) The query to perform on the Solr index.
+* `fl`: (Mandatory) The list of fields to return.
+* `sort`: (Mandatory) The sort criteria.
+* `zkHost`: Only needs to be defined if the collection being searched is found in a different zkHost than the local stream handler.
+* `partitionKeys`: Comma delimited list of keys to partition the search results by. To be used with the parallel function for parallelizing operations across worker nodes. See the <<stream-decorator-reference.adoc#parallel,parallel>> function for details.
+
+=== shuffle Syntax
+
+[source,text]
+----
+shuffle(collection1,
+ q="*:*",
+ fl="id,a_s,a_i,a_f",
+ sort="a_f asc, a_i asc")
+----
+
+== stats
+
+The `stats` function gathers simple aggregations for a search result set. The stats function does not support rollups over buckets, so the stats stream always returns a single tuple with the rolled up stats. Under the covers the stats function pushes down the generation of the stats into the search engine using the StatsComponent. The stats function currently supports the following metrics: `count(*)`, `sum()`, `avg()`, `min()`, and `max()`.
+
+=== stats Parameters
+
+* `collection`: (Mandatory) Collection the stats will be aggregated from.
+* `q`: (Mandatory) The query to build the aggregations from.
+* `metrics`: (Mandatory) The metrics to include in the result tuple. Current supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)` and `count(*)`
+
+=== stats Syntax
+
+[source,text]
+----
+stats(collection1,
+ q=*:*,
+ sum(a_i),
+ sum(a_f),
+ min(a_i),
+ min(a_f),
+ max(a_i),
+ max(a_f),
+ avg(a_i),
+ avg(a_f),
+ count(*))
+----
+
+== timeseries
+
+The `timeseries` function builds a time series aggregation. Under the covers the `timeseries` function uses the
+JSON Facet API as its high performance aggregation engine.
+
+=== timeseries Parameters
+
+* `collection`: (Mandatory) Collection the stats will be aggregated from.
+* `q`: (Mandatory) The query to build the aggregations from.
+* `field`: (Mandatory) The date field for the time series.
+* `start`: (Mandatory) The start of the time series expressed in Solr date or date math syntax.
+* `end`: (Mandatory) The end of the time series expressed in Solr date or date math syntax.
+* `gap`: (Mandatory) The time gap between time series aggregation points expressed in Solr date math syntax.
+* `format`: (Optional) Date template to format the date field in the output tuples. Formatting is performed by Java's SimpleDateFormat class.
+* `metrics`: (Mandatory) The metrics to include in the result tuple. Current supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)` and `count(*)`
+
+=== timeseries Syntax
+
+[source,text]
+----
+timeseries(collection1,
+ q=*:*,
+ field="rec_dt"
+ start="NOW-30DAYS",
+ end="NOW",
+ gap="+1DAY",
+ format="YYYY-MM-dd",
+ sum(a_i),
+ max(a_i),
+ max(a_f),
+ avg(a_i),
+ avg(a_f),
+ count(*))
+----
+
+== train
+
+The `train` function trains a Logistic Regression text classifier on a training set stored in a SolrCloud collection. It uses a parallel iterative, batch Gradient Descent approach to train the model. The training algorithm is embedded inside Solr so with each iteration only the model is streamed across the network.
+
+The `train` function wraps a <<features,features>> function which provides the terms and inverse document frequency (IDF) used to train the model. The `train` function operates over the same training set as the `features` function, which includes both positive and negative examples of the class.
+
+With each iteration the `train` function emits a tuple with the model. The model contains the feature terms, weights, and the confusion matrix for the model. The optimized model can then be used to classify documents based on their feature terms.
+
+=== train Parameters
+
+* `collection`: (Mandatory) Collection that holds the training set
+* `q`: (Mandatory) The query that defines the training set. The IDF for the features will be generated on the
+* `name`: (Mandatory) The name of model. This can be used to retrieve the model if they stored in a Solr Cloud collection.
+* `field`: (Mandatory) The text field to extract the features from.
+* `outcome`: (Mandatory) The field that defines the class, positive or negative
+* `maxIterations`: (Mandatory) How many training iterations to perform.
+* `positiveLabel`: (defaults to 1) The value in the outcome field that defines a positive outcome.
+
+=== train Syntax
+
+[source,text]
+----
+train(collection1,
+ features(collection1, q="*:*", featureSet="first", field="body", outcome="out_i", numTerms=250),
+ q="*:*",
+ name="model1",
+ field="body",
+ outcome="out_i",
+ maxIterations=100)
+----
+
+== topic
+
+The `topic` function provides publish/subscribe messaging capabilities built on top of SolrCloud. The topic function allows users to subscribe to a query. The function then provides one-time delivery of new or updated documents that match the topic query. The initial call to the topic function establishes the checkpoints for the specific topic ID. Subsequent calls to the same topic ID will return documents added or updated after the initial checkpoint. Each run of the topic query updates the checkpoints for the topic ID. Setting the initialCheckpoint parameter to 0 will cause the topic to process all documents in the index that match the topic query.
+
+[WARNING]
+====
+The topic function should be considered in beta until https://issues.apache.org/jira/browse/SOLR-8709[SOLR-8709] is committed and released.
+====
+
+=== topic Parameters
+
+* `checkpointCollection`: (Mandatory) The collection where the topic checkpoints are stored.
+* `collection`: (Mandatory) The collection that the topic query will be run on.
+* `id`: (Mandatory) The unique ID for the topic. The checkpoints will be saved under this id.
+* `q`: (Mandatory) The topic query.
+* `fl`: (Mandatory) The field list returned by the topic function.
+* `initialCheckpoint`: (Optional) Sets the initial Solr `\_version_` number to start reading from the queue. If not set, it defaults to the highest version in the index. Setting to 0 will process all records that match query in the index.
+
+=== topic Syntax
+
+[source,text]
+----
+topic(checkpointCollection,
+ collection,
+ id="uniqueId",
+ q="topic query",
+ fl="id, name, country")
+----
+
+== tuple
+
+The `tuple` function emits a single Tuple with name/value pairs. The values can be set to variables assigned in a `let` expression, literals, Stream Evaluators or
+Stream Expressions. In the case of Stream Evaluators the tuple will output the return value from the evaluator.
+This could be a numeric, list or map. If a value is set to a Stream Expression, the `tuple` function will flatten
+the tuple stream from the Stream Expression into a list of Tuples.
+
+=== tuple Parameters
+
+* name=value pairs
+
+=== tuple Syntax
+
+[source,text]
+----
+tuple(a=add(1,1),
+ b=search(collection1, q="cat:a", fl="a, b, c", sort="a desc"))
+----
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9beafb61/solr/solr-ref-guide/src/stream-sources.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/stream-sources.adoc b/solr/solr-ref-guide/src/stream-sources.adoc
deleted file mode 100644
index 2b59edf..0000000
--- a/solr/solr-ref-guide/src/stream-sources.adoc
+++ /dev/null
@@ -1,538 +0,0 @@
-= Stream Source Reference
-:page-shortname: stream-sources
-:page-permalink: stream-sources.html
-:page-tocclass: right
-:page-toclevels: 1
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-
-== search
-
-The `search` function searches a SolrCloud collection and emits a stream of tuples that match the query. This is very similar to a standard Solr query, and uses many of the same parameters.
-
-This expression allows you to specify a request hander using the `qt` parameter. By default, the `/select` handler is used. The `/select` handler can be used for simple rapid prototyping of expressions. For production, however, you will most likely want to use the `/export` handler which is designed to `sort` and `export` entire result sets. The `/export` handler is not used by default because it has stricter requirements then the `/select` handler so it's not as easy to get started working with. To read more about the `/export` handler requirements review the section <<exporting-result-sets.adoc#exporting-result-sets,Exporting Result Sets>>.
-
-=== search Parameters
-
-* `collection`: (Mandatory) the collection being searched.
-* `q`: (Mandatory) The query to perform on the Solr index.
-* `fl`: (Mandatory) The list of fields to return.
-* `sort`: (Mandatory) The sort criteria.
-* `zkHost`: Only needs to be defined if the collection being searched is found in a different zkHost than the local stream handler.
-* `qt`: Specifies the query type, or request handler, to use. Set this to `/export` to work with large result sets. The default is `/select`.
-* `rows`: (Mandatory with the `/select` handler) The rows parameter specifies how many rows to return. This parameter is only needed with the `/select` handler (which is the default) since the `/export` handler always returns all rows.
-* `partitionKeys`: Comma delimited list of keys to partition the search results by. To be used with the parallel function for parallelizing operations across worker nodes. See the <<stream-decorators.adoc#parallel,parallel>> function for details.
-
-=== search Syntax
-
-[source,text]
-----
-expr=search(collection1,
- zkHost="localhost:9983",
- qt="/export",
- q="*:*",
- fl="id,a_s,a_i,a_f",
- sort="a_f asc, a_i asc")
-----
-
-== jdbc
-
-The `jdbc` function searches a JDBC datasource and emits a stream of tuples representing the JDBC result set. Each row in the result set is translated into a tuple and each tuple contains all the cell values for that row.
-
-=== jdbc Parameters
-
-* `connection`: (Mandatory) JDBC formatted connection string to whatever driver you are using.
-* `sql`: (Mandatory) query to pass off to the JDBC endpoint
-* `sort`: (Mandatory) The sort criteria indicating how the data coming out of the JDBC stream is sorted
-* `driver`: The name of the JDBC driver used for the connection. If provided then the driver class will attempt to be loaded into the JVM. If not provided then it is assumed that the driver is already loaded into the JVM. Some drivers require explicit loading so this option is provided.
-* `[driverProperty]`: One or more properties to pass to the JDBC driver during connection. The format is `propertyName="propertyValue"`. You can provide as many of these properties as you'd like and they will all be passed to the connection.
-
-=== Connections and Drivers
-
-Because some JDBC drivers require explicit loading the `driver` parameter can be used to provide the driver class name. If provided, then during stream construction the driver will be loaded. If the driver cannot be loaded because the class is not found on the classpath, then stream construction will fail.
-
-When the JDBC stream is opened it will validate that a driver can be found for the provided connection string. If a driver cannot be found (because it hasn't been loaded) then the open will fail.
-
-=== Datatypes
-
-Due to the inherent differences in datatypes across JDBC sources the following datatypes are supported. The table indicates what Java type will be used for a given JDBC type. Types marked as requiring conversion will go through a conversion for each value of that type. For performance reasons the cell data types are only considered when the stream is opened as this is when the converters are created.
-
-[width="100%",options="header",]
-|===
-|JDBC Type |Java Type |Requires Conversion
-|String |String |No
-|Short |Long |Yes
-|Integer |Long |Yes
-|Long |Long |No
-|Float |Double |Yes
-|Double |Double |No
-|Boolean |Boolean |No
-|===
-
-=== jdbc Syntax
-
-A basic `jdbc` expression:
-
-[source,text]
-----
-jdbc(
- connection="jdbc:hsqldb:mem:.",
- sql="select NAME, ADDRESS, EMAIL, AGE from PEOPLE where AGE > 25 order by AGE, NAME DESC",
- sort="AGE asc, NAME desc",
- driver="org.hsqldb.jdbcDriver"
-)
-----
-
-A `jdbc` expression that passes a property to the driver:
-
-[source,text]
-----
-// get_column_name is a property to pass to the hsqldb driver
-jdbc(
- connection="jdbc:hsqldb:mem:.",
- sql="select NAME as FIRST_NAME, ADDRESS, EMAIL, AGE from PEOPLE where AGE > 25 order by AGE, NAME DESC",
- sort="AGE asc, NAME desc",
- driver="org.hsqldb.jdbcDriver",
- get_column_name="false"
-)
-----
-
-== echo
-
-The `echo` function returns a single Tuple echoing its text parameter. `Echo` is the simplest stream source designed to provide text
-to a text analyzing stream decorator.
-
-=== echo Syntax
-
-[source,text]
-----
-echo("Hello world")
-----
-
-== facet
-
-The `facet` function provides aggregations that are rolled up over buckets. Under the covers the facet function pushes down the aggregation into the search engine using Solr's JSON Facet API. This provides sub-second performance for many use cases. The facet function is appropriate for use with a low to moderate number of distinct values in the bucket fields. To support high cardinality aggregations see the rollup function.
-
-=== facet Parameters
-
-* `collection`: (Mandatory) Collection the facets will be aggregated from.
-* `q`: (Mandatory) The query to build the aggregations from.
-* `buckets`: (Mandatory) Comma separated list of fields to rollup over. The comma separated list represents the dimensions in a multi-dimensional rollup.
-* `bucketSorts`: Comma separated list of sorts to apply to each dimension in the buckets parameters. Sorts can be on the computed metrics or on the bucket values.
-* `bucketSizeLimit`: The number of buckets to include. This value is applied to each dimension.
-* `metrics`: List of metrics to compute for the buckets. Currently supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)`, `count(*)`.
-
-=== facet Syntax
-
-Example 1:
-
-[source,text]
-----
-facet(collection1,
- q="*:*",
- buckets="a_s",
- bucketSorts="sum(a_i) desc",
- bucketSizeLimit=100,
- sum(a_i),
- sum(a_f),
- min(a_i),
- min(a_f),
- max(a_i),
- max(a_f),
- avg(a_i),
- avg(a_f),
- count(*))
-----
-
-The example above shows a facet function with rollups over a single bucket, where the buckets are returned in descending order by the calculated value of the `sum(a_i)` metric.
-
-Example 2:
-
-[source,text]
-----
-facet(collection1,
- q="*:*",
- buckets="year_i, month_i, day_i",
- bucketSorts="year_i desc, month_i desc, day_i desc",
- bucketSizeLimit=100,
- sum(a_i),
- sum(a_f),
- min(a_i),
- min(a_f),
- max(a_i),
- max(a_f),
- avg(a_i),
- avg(a_f),
- count(*))
-----
-
-The example above shows a facet function with rollups over three buckets, where the buckets are returned in descending order by bucket value.
-
-== features
-
-The `features` function extracts the key terms from a text field in a classification training set stored in a SolrCloud collection. It uses an algorithm known as * Information Gain* , to select the important terms from the training set. The `features` function was designed to work specifically with the <<train,train>> function, which uses the extracted features to train a text classifier.
-
-The `features` function is designed to work with a training set that provides both positive and negative examples of a class. It emits a tuple for each feature term that is extracted along with the inverse document frequency (IDF) for the term in the training set.
-
-The `features` function uses a query to select the training set from a collection. The IDF for each selected feature is calculated relative to the training set matching the query. This allows multiple training sets to be stored in the same SolrCloud collection without polluting the IDF across training sets.
-
-=== features Parameters
-
-* `collection`: (Mandatory) The collection that holds the training set
-* `q`: (Mandatory) The query that defines the training set. The IDF for the features will be generated specific to the result set matching the query.
-* `featureSet`: (Mandatory) The name of the feature set. This can be used to retrieve the features if they are stored in a SolrCloud collection.
-* `field`: (Mandatory) The text field to extract the features from.
-* `outcome`: (Mandatory) The field that defines the class, positive or negative
-* `numTerms`: (Mandatory) How many feature terms to extract.
-* `positiveLabel`: (defaults to 1) The value in the outcome field that defines a postive outcome.
-
-=== features Syntax
-
-[source,text]
-----
-features(collection1,
- q="*:*",
- featureSet="features1",
- field="body",
- outcome="out_i",
- numTerms=250)
-----
-
-== nodes
-
-The `nodes` function provides breadth-first graph traversal. For details, see the section <<graph-traversal.adoc#graph-traversal,Graph Traversal>>.
-
-== knn
-
-The `knn` function returns the K nearest neighbors for a document based on text similarity. Under the covers the `knn` function
-use the More Like This query parser plugin.
-
-=== knn Parameters
-
-* `collection`: (Mandatory) The collection to perform the search in.
-* `id`: (Mandatory) The id of the source document to begin the knn search from.
-* `qf`: (Mandatory) The query field used to compare documents.
-* `k`: (Mandatory) The number of nearest neighbors to return.
-* `fl`: (Mandatory) The field list to return.
-* `mintf`: (Optional) The minimum numer of occurrences of the term in the source document to be inlcued in the search.
-* `maxtf`: (Optional) The maximum numer of occurrences of the term in the source document to be inlcued in the search.
-* `mindf`: (Optional) The minimum numer of occurrences in the corpus to be inlcued in the search.
-* `maxdf`: (Optional) The maximum numer of occurrences in the corpus to be inlcued in the search.
-* `minwl`: (Optional) The minimum world length of to be inlcued in the search.
-* `maxwl`: (Optional) The maximum world length of to be inlcued in the search.
-
-=== knn Syntax
-
-[source,text]
-----
-knn(collection1,
- id="doc1",
- qf="text_field",
- k="10",
- fl="id, title",
- mintf="3",
- maxdf="10000000")
-----
-
-== model
-
-The `model` function retrieves and caches logistic regression text classification models that are stored in a SolrCloud collection. The `model` function is designed to work with models that are created by the <<train,train function>>, but can also be used to retrieve text classification models trained outside of Solr, as long as they conform to the specified format. After the model is retrieved it can be used by the <<stream-decorators.adoc#classify,classify function>> to classify documents.
-
-A single model tuple is fetched and returned based on the *id* parameter. The model is retrieved by matching the *id* parameter with a model name in the index. If more then one iteration of the named model is stored in the index, the highest iteration is selected.
-
-=== Caching with model
-
-The `model` function has an internal LRU (least-recently-used) cache so models do not have to be retrieved with each invocation of the `model` function. The time to cache for each model ID can be passed as a parameter to the function call. Retrieving a cached model does not reset the time for expiring the model ID in the cache.
-
-=== Model Storage
-
-The storage format of the models in Solr is below. The `train` function outputs the format below so you only need to know schema details if you plan to use the `model` function with logistic regression models trained outside of Solr.
-
-* `name_s` (Single value, String, Stored): The name of the model.
-* `iteration_i` (Single value, Integer, Stored): The iteration number of the model. Solr can store all iterations of the models generated by the train function.
-* `terms_ss` (Multi value, String, Stored: The array of terms/features of the model.
-* `weights_ds` (Multi value, double, Stored): The array of term weights. Each weight corresponds by array index to a term.
-* `idfs_ds` (Multi value, double, Stored): The array of term IDFs (Inverse document frequency). Each IDF corresponds by array index to a term.
-
-=== model Parameters
-
-* `collection`: (Mandatory) The collection where the model is stored.
-* `id`: (Mandatory) The id/name of the model. The model function always returns one model. If there are multiple iterations of the name, the highest iteration is returned.
-* `cacheMillis`: (Optional) The amount of time to cache the model in the LRU cache.
-
-=== model Syntax
-
-[source,text]
-----
-model(modelCollection,
- id="myModel"
- cacheMillis="200000")
-----
-
-== random
-
-The `random` function searches a SolrCloud collection and emits a pseudo-random set of results that match the query. Each invocation of random will return a different pseudo-random result set.
-
-=== random Parameters
-
-* `collection`: (Mandatory) The collection the stats will be aggregated from.
-* `q`: (Mandatory) The query to build the aggregations from.
-* `rows`: (Mandatory) The number of pseudo-random results to return.
-* fl: (Mandatory) The field list to return.
-* `fq`: (Optional) Filter query
-
-=== random Syntax
-
-[source,text]
-----
-random(baskets,
- q="productID:productX",
- rows="100",
- fl="basketID")
-----
-
-In the example above the `random` function is searching the baskets collections for all rows where "productID:productX". It will return 100 pseudo-random results. The field list returned is the basketID.
-
-== significantTerms
-
-The `significantTerms` function queries a SolrCloud collection, but instead of returning documents, it returns significant terms found in documents in the result set. The `significantTerms` function scores terms based on how frequently they appear in the result set and how rarely they appear in the entire corpus. The `significantTerms` function emits a tuple for each term which contains the term, the score, the foreground count and the background count. The foreground count is how many documents the term appears in in the result set. The background count is how many documents the term appears in in the entire corpus. The foreground and background counts are global for the collection.
-
-=== significantTerms Parameters
-
-* `collection`: (Mandatory) The collection that the function is run on.
-* `q`: (Mandatory) The query that describes the foreground document set.
-* `field`: (Mandatory) The field to extract the terms from.
-* `limit`: (Optional, Default 20) The max number of terms to return.
-* `minDocFreq`: (Optional, Defaults to 5 documents) The minimum number of documents the term must appear in on a shard. This is a float value. If greater then 1.0 then it's considered the absolute number of documents. If less then 1.0 it's treated as a percentage of documents.
-* `maxDocFreq`: (Optional, Defaults to 30% of documents) The maximum number of documents the term can appear in on a shard. This is a float value. If greater then 1.0 then it's considered the absolute number of documents. If less then 1.0 it's treated as a percentage of documents.
-* `minTermLength`: (Optional, Default 4) The minimum length of the term to be considered significant.
-
-=== significantTerms Syntax
-
-[source,text]
-----
-significantTerms(collection1,
- q="body:Solr",
- field="author",
- limit="50",
- minDocFreq="10",
- maxDocFreq=".20",
- minTermLength="5")
-----
-
-In the example above the `significantTerms` function is querying `collection1` and returning at most 50 significant terms from the `authors` field that appear in 10 or more documents but not more then 20% of the corpus.
-
-== shortestPath
-
-The `shortestPath` function is an implementation of a shortest path graph traversal. The `shortestPath` function performs an iterative breadth-first search through an unweighted graph to find the shortest paths between two nodes in a graph. The `shortestPath` function emits a tuple for each path found. Each tuple emitted will contain a `path` key which points to a `List` of nodeIDs comprising the path.
-
-=== shortestPath Parameters
-
-* `collection`: (Mandatory) The collection that the topic query will be run on.
-* `from`: (Mandatory) The nodeID to start the search from
-* `to`: (Mandatory) The nodeID to end the search at
-* `edge`: (Mandatory) Syntax: `from_field=to_field`. The `from_field` defines which field to search from. The `to_field` defines which field to search to. See example below for a detailed explanation.
-* `threads`: (Optional : Default 6) The number of threads used to perform the partitioned join in the traversal.
-* `partitionSize`: (Optional : Default 250) The number of nodes in each partition of the join.
-* `fq`: (Optional) Filter query
-* `maxDepth`: (Mandatory) Limits to the search to a maximum depth in the graph.
-
-=== shortestPath Syntax
-
-[source,text]
-----
-shortestPath(collection,
- from="john@company.com",
- to="jane@company.com",
- edge="from_address=to_address",
- threads="6",
- partitionSize="300",
- fq="limiting query",
- maxDepth="4")
-----
-
-The expression above performs a breadth-first search to find the shortest paths in an unweighted, directed graph.
-
-The search starts from the nodeID "\john@company.com" in the `from_address` field and searches for the nodeID "\jane@company.com" in the `to_address` field. This search is performed iteratively until the `maxDepth` has been reached. Each level in the traversal is implemented as a parallel partitioned nested loop join across the entire collection. The `threads` parameter controls the number of threads performing the join at each level, while the `partitionSize` parameter controls the of number of nodes in each join partition. The `maxDepth` parameter controls the number of levels to traverse. `fq` is a limiting query applied to each level in the traversal.
-
-== shuffle
-
-The `shuffle` expression sorts and exports entire result sets. The `shuffle` expression is similar to the `search` expression except that
-under the covers `shuffle` always uses the /export handler. The `shuffle` expression is designed to be combined with the relational algebra
-decorators that require complete, sorted result sets. Shuffled result sets can be partitioned across worker nodes with the parallel
-stream decorator to perform parallel relational algebra. When used in parallel mode the partitionKeys parameter must be provided.
-
-=== shuffle Parameters
-
-* `collection`: (Mandatory) the collection being searched.
-* `q`: (Mandatory) The query to perform on the Solr index.
-* `fl`: (Mandatory) The list of fields to return.
-* `sort`: (Mandatory) The sort criteria.
-* `zkHost`: Only needs to be defined if the collection being searched is found in a different zkHost than the local stream handler.
-* `partitionKeys`: Comma delimited list of keys to partition the search results by. To be used with the parallel function for parallelizing operations across worker nodes. See the <<stream-decorators.adoc#parallel,parallel>> function for details.
-
-=== shuffle Syntax
-
-[source,text]
-----
-shuffle(collection1,
- q="*:*",
- fl="id,a_s,a_i,a_f",
- sort="a_f asc, a_i asc")
-----
-
-== stats
-
-The `stats` function gathers simple aggregations for a search result set. The stats function does not support rollups over buckets, so the stats stream always returns a single tuple with the rolled up stats. Under the covers the stats function pushes down the generation of the stats into the search engine using the StatsComponent. The stats function currently supports the following metrics: `count(*)`, `sum()`, `avg()`, `min()`, and `max()`.
-
-=== stats Parameters
-
-* `collection`: (Mandatory) Collection the stats will be aggregated from.
-* `q`: (Mandatory) The query to build the aggregations from.
-* `metrics`: (Mandatory) The metrics to include in the result tuple. Current supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)` and `count(*)`
-
-=== stats Syntax
-
-[source,text]
-----
-stats(collection1,
- q=*:*,
- sum(a_i),
- sum(a_f),
- min(a_i),
- min(a_f),
- max(a_i),
- max(a_f),
- avg(a_i),
- avg(a_f),
- count(*))
-----
-
-== timeseries
-
-The `timeseries` function builds a time series aggregation. Under the covers the `timeseries` function uses the
-JSON Facet API as its high performance aggregation engine.
-
-=== timeseries Parameters
-
-* `collection`: (Mandatory) Collection the stats will be aggregated from.
-* `q`: (Mandatory) The query to build the aggregations from.
-* `field`: (Mandatory) The date field for the time series.
-* `start`: (Mandatory) The start of the time series expressed in Solr date or date math syntax.
-* `end`: (Mandatory) The end of the time series expressed in Solr date or date math syntax.
-* `gap`: (Mandatory) The time gap between time series aggregation points expressed in Solr date math syntax.
-* `format`: (Optional) Date template to format the date field in the output tuples. Formatting is performed by Java's SimpleDateFormat class.
-* `metrics`: (Mandatory) The metrics to include in the result tuple. Current supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)` and `count(*)`
-
-=== timeseries Syntax
-
-[source,text]
-----
-timeseries(collection1,
- q=*:*,
- field="rec_dt"
- start="NOW-30DAYS",
- end="NOW",
- gap="+1DAY",
- format="YYYY-MM-dd",
- sum(a_i),
- max(a_i),
- max(a_f),
- avg(a_i),
- avg(a_f),
- count(*))
-----
-
-== train
-
-The `train` function trains a Logistic Regression text classifier on a training set stored in a SolrCloud collection. It uses a parallel iterative, batch Gradient Descent approach to train the model. The training algorithm is embedded inside Solr so with each iteration only the model is streamed across the network.
-
-The `train` function wraps a <<features,features>> function which provides the terms and inverse document frequency (IDF) used to train the model. The `train` function operates over the same training set as the `features` function, which includes both positive and negative examples of the class.
-
-With each iteration the `train` function emits a tuple with the model. The model contains the feature terms, weights, and the confusion matrix for the model. The optimized model can then be used to classify documents based on their feature terms.
-
-=== train Parameters
-
-* `collection`: (Mandatory) Collection that holds the training set
-* `q`: (Mandatory) The query that defines the training set. The IDF for the features will be generated on the
-* `name`: (Mandatory) The name of model. This can be used to retrieve the model if they stored in a Solr Cloud collection.
-* `field`: (Mandatory) The text field to extract the features from.
-* `outcome`: (Mandatory) The field that defines the class, positive or negative
-* `maxIterations`: (Mandatory) How many training iterations to perform.
-* `positiveLabel`: (defaults to 1) The value in the outcome field that defines a positive outcome.
-
-=== train Syntax
-
-[source,text]
-----
-train(collection1,
- features(collection1, q="*:*", featureSet="first", field="body", outcome="out_i", numTerms=250),
- q="*:*",
- name="model1",
- field="body",
- outcome="out_i",
- maxIterations=100)
-----
-
-== topic
-
-The `topic` function provides publish/subscribe messaging capabilities built on top of SolrCloud. The topic function allows users to subscribe to a query. The function then provides one-time delivery of new or updated documents that match the topic query. The initial call to the topic function establishes the checkpoints for the specific topic ID. Subsequent calls to the same topic ID will return documents added or updated after the initial checkpoint. Each run of the topic query updates the checkpoints for the topic ID. Setting the initialCheckpoint parameter to 0 will cause the topic to process all documents in the index that match the topic query.
-
-[WARNING]
-====
-The topic function should be considered in beta until https://issues.apache.org/jira/browse/SOLR-8709[SOLR-8709] is committed and released.
-====
-
-=== topic Parameters
-
-* `checkpointCollection`: (Mandatory) The collection where the topic checkpoints are stored.
-* `collection`: (Mandatory) The collection that the topic query will be run on.
-* `id`: (Mandatory) The unique ID for the topic. The checkpoints will be saved under this id.
-* `q`: (Mandatory) The topic query.
-* `fl`: (Mandatory) The field list returned by the topic function.
-* `initialCheckpoint`: (Optional) Sets the initial Solr `\_version_` number to start reading from the queue. If not set, it defaults to the highest version in the index. Setting to 0 will process all records that match query in the index.
-
-=== topic Syntax
-
-[source,text]
-----
-topic(checkpointCollection,
- collection,
- id="uniqueId",
- q="topic query",
- fl="id, name, country")
-----
-
-== tuple
-
-The `tuple` function emits a single Tuple with name/value pairs. The values can be set to variables assigned in a `let` expression, literals, Stream Evaluators or
-Stream Expressions. In the case of Stream Evaluators the tuple will output the return value from the evaluator.
-This could be a numeric, list or map. If a value is set to a Stream Expression, the `tuple` function will flatten
-the tuple stream from the Stream Expression into a list of Tuples.
-
-=== tuple Parameters
-
-* name=value pairs
-
-=== tuple Syntax
-
-[source,text]
-----
-tuple(a=add(1,1),
- b=search(collection1, q="cat:a", fl="a, b, c", sort="a desc"))
-----
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9beafb61/solr/solr-ref-guide/src/streaming-expressions.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/streaming-expressions.adoc b/solr/solr-ref-guide/src/streaming-expressions.adoc
index 6d13573..95402ef 100644
--- a/solr/solr-ref-guide/src/streaming-expressions.adoc
+++ b/solr/solr-ref-guide/src/streaming-expressions.adoc
@@ -1,7 +1,7 @@
= Streaming Expressions
:page-shortname: streaming-expressions
:page-permalink: streaming-expressions.html
-:page-children: stream-sources, stream-decorators, stream-evaluators, statistical-programming, graph-traversal
+:page-children: stream-source-reference, stream-decorator-reference, stream-evaluator-reference, statistical-programming, graph-traversal
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
@@ -121,12 +121,12 @@ Because streaming expressions relies on the `/export` handler, many of the field
Stream sources originate streams. The most commonly used one of these is `search`, which does a query.
-A full reference to all available source expressions is available in <<stream-sources.adoc#stream-sources,Stream Sources>>.
+A full reference to all available source expressions is available in <<stream-source-reference.adoc#stream-source-reference,Stream Source Reference>>.
=== About Stream Decorators
Stream decorators wrap other stream functions or perform operations on a stream.
-A full reference to all available decorator expressions is available in <<stream-decorators.adoc#stream-decorators,Stream Decorators>>.
+A full reference to all available decorator expressions is available in <<stream-decorator-reference.adoc#stream-decorator-reference,Stream Decorator Reference>>.
=== About Stream Evaluators
@@ -141,4 +141,4 @@ In cases where you want to use raw values as part of an evaluation you will need
If you wish to use a raw string as part of an evaluation, you will want to consider using the `raw(string)` evaluator. This will always return the raw value, no matter what is entered.
-A full reference to all available evaluator expressions is available in <<stream-evaluators.adoc#stream-evaluators,Stream Evaluators>>.
+A full reference to all available evaluator expressions is available in <<stream-evaluator-reference.adoc#stream-evaluator-reference,Stream Evaluator Reference>>.
[27/50] [abbrv] lucene-solr:feature/autoscaling: LUCENE-7978: Add
some more documentation about setting up build environment;
merge branch 'LUCENE-7978' of https://github.com/antonmry/lucene-solr
Posted by no...@apache.org.
LUCENE-7978: Add some more documentation about setting up build environment; merge branch 'LUCENE-7978' of https://github.com/antonmry/lucene-solr
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/f1bb9147
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/f1bb9147
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/f1bb9147
Branch: refs/heads/feature/autoscaling
Commit: f1bb91478449ee7613a28feb596f497dbcd82042
Parents: eaa5586 e148574
Author: Uwe Schindler <us...@apache.org>
Authored: Fri Sep 29 14:53:56 2017 +0200
Committer: Uwe Schindler <us...@apache.org>
Committed: Fri Sep 29 14:53:56 2017 +0200
----------------------------------------------------------------------
README.md | 8 ++++++++
lucene/CHANGES.txt | 3 +++
2 files changed, 11 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/f1bb9147/README.md
----------------------------------------------------------------------
diff --cc README.md
index 7181297,50da250..0dac902
--- a/README.md
+++ b/README.md
@@@ -41,6 -41,12 +41,14 @@@ Lucene and Solr, run
`ant compile`
-If you see an error about Ivy missing when invoking ant (e.g., `.ant/lib does not exist`), run `ant ivy-bootstrap` and retry.
++If you see an error about Ivy missing while invoking Ant (e.g., `.ant/lib does
++not exist`), run `ant ivy-bootstrap` and retry.
+
-Sometimes you may face issues with Ivy (e.g., instance a wrongly downloaded artifact). Cleaning up the Ivy cache and retrying is a workaround for most of such issues:
++Sometimes you may face issues with Ivy (e.g., an incompletely downloaded artifact).
++Cleaning up the Ivy cache and retrying is a workaround for most of such issues:
+
+ `rm -rf ~/.ivy2/cache`
+
The Solr server can then be packaged and prepared for startup by running the
following command from the `solr/` directory:
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/f1bb9147/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --cc lucene/CHANGES.txt
index ffc901c,ffc901c..ae61d86
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@@ -92,6 -92,6 +92,9 @@@ Othe
* LUCENE-7933: LongBitSet now validates the numBits parameter (Won
Jonghoon, Mike McCandless)
++* LUCENE-7978: Add some more documentation about setting up build
++ environment. (Anton R. Yuste via Uwe Schindler)
++
======================= Lucene 7.0.0 =======================
New Features
[43/50] [abbrv] lucene-solr:feature/autoscaling: LUCENE-7974:
nearest() method returning NearestHit should be private,
and NearestHit class should be package private
Posted by no...@apache.org.
LUCENE-7974: nearest() method returning NearestHit should be private, and NearestHit class should be package private
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/74050a3f
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/74050a3f
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/74050a3f
Branch: refs/heads/feature/autoscaling
Commit: 74050a3f159eca393ccb8e0b28c4fc4f974f6d5e
Parents: 73f3403
Author: Steve Rowe <sa...@gmail.com>
Authored: Mon Oct 2 10:39:31 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Mon Oct 2 10:39:31 2017 -0400
----------------------------------------------------------------------
.../org/apache/lucene/document/FloatPointNearestNeighbor.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/74050a3f/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
----------------------------------------------------------------------
diff --git a/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java b/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
index b7859d0..90a80b6 100644
--- a/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
+++ b/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
@@ -194,7 +194,7 @@ public class FloatPointNearestNeighbor {
}
/** Holds one hit from {@link FloatPointNearestNeighbor#nearest} */
- public static class NearestHit {
+ static class NearestHit {
public int docID;
public double distanceSquared;
@@ -204,7 +204,7 @@ public class FloatPointNearestNeighbor {
}
}
- public static NearestHit[] nearest(List<BKDReader> readers, List<Bits> liveDocs, List<Integer> docBases, final int topN, float[] origin) throws IOException {
+ private static NearestHit[] nearest(List<BKDReader> readers, List<Bits> liveDocs, List<Integer> docBases, final int topN, float[] origin) throws IOException {
// System.out.println("NEAREST: readers=" + readers + " liveDocs=" + liveDocs + " origin: " + Arrays.toString(origin));
[22/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11278: Fix a
race condition in the CDCR bootstrap process which could lead to bootstraps
cancelling itself
Posted by no...@apache.org.
SOLR-11278: Fix a race condition in the CDCR bootstrap process which could lead to bootstraps cancelling itself
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/26677ab2
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/26677ab2
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/26677ab2
Branch: refs/heads/feature/autoscaling
Commit: 26677ab2b041a074fd984457677cbea8b58363ab
Parents: 5e840c9
Author: Varun Thacker <va...@apache.org>
Authored: Wed Sep 27 14:34:45 2017 -0700
Committer: Varun Thacker <va...@apache.org>
Committed: Wed Sep 27 14:34:45 2017 -0700
----------------------------------------------------------------------
solr/CHANGES.txt | 3 +++
.../java/org/apache/solr/handler/CdcrRequestHandler.java | 9 ++++++++-
2 files changed, 11 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/26677ab2/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 2b8a9c5..13cee84 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -153,6 +153,9 @@ Bug Fixes
* SOLR-11224: SolrStream.close can hit an NPE (Erick Erickson)
+* SOLR-11278: Fix a race condition in the CDCR bootstrap process which could lead to bootstraps cancelling itself
+ (Amrit Sarkar, shalin, Varun Thacker)
+
Optimizations
----------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/26677ab2/solr/core/src/java/org/apache/solr/handler/CdcrRequestHandler.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/CdcrRequestHandler.java b/solr/core/src/java/org/apache/solr/handler/CdcrRequestHandler.java
index caf88f4..430237e 100644
--- a/solr/core/src/java/org/apache/solr/handler/CdcrRequestHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/CdcrRequestHandler.java
@@ -26,10 +26,12 @@ import java.util.List;
import java.util.Map;
import java.util.concurrent.Callable;
import java.util.concurrent.CancellationException;
+import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.Lock;
import org.apache.solr.client.solrj.SolrRequest;
@@ -617,7 +619,7 @@ public class CdcrRequestHandler extends RequestHandlerBase implements SolrCoreAw
rsp.add(CdcrParams.ERRORS, hosts);
}
- private void handleBootstrapAction(SolrQueryRequest req, SolrQueryResponse rsp) throws IOException, SolrServerException {
+ private void handleBootstrapAction(SolrQueryRequest req, SolrQueryResponse rsp) throws IOException, InterruptedException, SolrServerException {
String collectionName = core.getCoreDescriptor().getCloudDescriptor().getCollectionName();
String shard = core.getCoreDescriptor().getCloudDescriptor().getShardId();
if (!leaderStateManager.amILeader()) {
@@ -625,6 +627,7 @@ public class CdcrRequestHandler extends RequestHandlerBase implements SolrCoreAw
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Action " + CdcrParams.CdcrAction.BOOTSTRAP +
" sent to non-leader replica");
}
+ CountDownLatch latch = new CountDownLatch(1); // latch to make sure BOOTSTRAP_STATUS gives correct response
Runnable runnable = () -> {
Lock recoveryLock = req.getCore().getSolrCoreState().getRecoveryLock();
@@ -635,6 +638,7 @@ public class CdcrRequestHandler extends RequestHandlerBase implements SolrCoreAw
handleCancelBootstrap(req, rsp);
} else if (leaderStateManager.amILeader()) {
coreState.setCdcrBootstrapRunning(true);
+ latch.countDown(); // free the latch as current bootstrap is executing
//running.set(true);
String masterUrl = req.getParams().get(ReplicationHandler.MASTER_URL);
BootstrapCallable bootstrapCallable = new BootstrapCallable(masterUrl, core);
@@ -657,6 +661,8 @@ public class CdcrRequestHandler extends RequestHandlerBase implements SolrCoreAw
if (locked) {
coreState.setCdcrBootstrapRunning(false);
recoveryLock.unlock();
+ } else {
+ latch.countDown(); // free the latch as current bootstrap is executing
}
}
};
@@ -664,6 +670,7 @@ public class CdcrRequestHandler extends RequestHandlerBase implements SolrCoreAw
try {
core.getCoreContainer().getUpdateShardHandler().getUpdateExecutor().submit(runnable);
rsp.add(RESPONSE_STATUS, "submitted");
+ latch.await(10000, TimeUnit.MILLISECONDS); // put the latch for current bootstrap command
} catch (RejectedExecutionException ree) {
// no problem, we're probably shutting down
rsp.add(RESPONSE_STATUS, "failed");
[34/50] [abbrv] lucene-solr:feature/autoscaling: Add version 7.0.1
Posted by no...@apache.org.
Add version 7.0.1
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/dc47c6e3
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/dc47c6e3
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/dc47c6e3
Branch: refs/heads/feature/autoscaling
Commit: dc47c6e3502e1f8da99cce1f49f05355a395fa77
Parents: 39ba1c3
Author: Steve Rowe <sa...@gmail.com>
Authored: Sat Sep 30 06:58:43 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Sat Sep 30 06:58:43 2017 -0400
----------------------------------------------------------------------
lucene/CHANGES.txt | 3 +++
lucene/core/src/java/org/apache/lucene/util/Version.java | 7 +++++++
2 files changed, 10 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/dc47c6e3/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index ae61d86..16482e9 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -95,6 +95,9 @@ Other
* LUCENE-7978: Add some more documentation about setting up build
environment. (Anton R. Yuste via Uwe Schindler)
+======================= Lucene 7.0.1 =======================
+(No Changes)
+
======================= Lucene 7.0.0 =======================
New Features
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/dc47c6e3/lucene/core/src/java/org/apache/lucene/util/Version.java
----------------------------------------------------------------------
diff --git a/lucene/core/src/java/org/apache/lucene/util/Version.java b/lucene/core/src/java/org/apache/lucene/util/Version.java
index 0fe4f30..65134fb 100644
--- a/lucene/core/src/java/org/apache/lucene/util/Version.java
+++ b/lucene/core/src/java/org/apache/lucene/util/Version.java
@@ -40,6 +40,13 @@ public final class Version {
public static final Version LUCENE_7_0_0 = new Version(7, 0, 0);
/**
+ * Match settings and bugs in Lucene's 7.0.1 release.
+ * @deprecated Use latest
+ */
+ @Deprecated
+ public static final Version LUCENE_7_0_1 = new Version(7, 0, 1);
+
+ /**
* Match settings and bugs in Lucene's 7.1.0 release.
* @deprecated Use latest
*/
[19/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11293: Fix
downgrade in performance from precommit
Posted by no...@apache.org.
SOLR-11293: Fix downgrade in performance from precommit
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/cc9c8670
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/cc9c8670
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/cc9c8670
Branch: refs/heads/feature/autoscaling
Commit: cc9c867083831cbacfdc76466800522fda05e1f7
Parents: 61d1e96
Author: Cao Manh Dat <da...@apache.org>
Authored: Wed Sep 27 22:46:55 2017 +0700
Committer: Cao Manh Dat <da...@apache.org>
Committed: Wed Sep 27 22:46:55 2017 +0700
----------------------------------------------------------------------
solr/CHANGES.txt | 2 +-
.../org/apache/solr/cloud/RecoveryStrategy.java | 1 +
.../apache/solr/cloud/ReplicateFromLeader.java | 7 ++-
.../org/apache/solr/cloud/ZkController.java | 2 +-
.../org/apache/solr/handler/IndexFetcher.java | 16 +++++--
.../apache/solr/handler/ReplicationHandler.java | 10 ++--
.../solr/handler/admin/PrepRecoveryOp.java | 33 -------------
.../java/org/apache/solr/update/UpdateLog.java | 12 +++--
.../org/apache/solr/cloud/TestTlogReplica.java | 50 ++++++++++++--------
9 files changed, 66 insertions(+), 67 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/cc9c8670/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 7cd3717..2b8a9c5 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -134,7 +134,7 @@ Bug Fixes
* SOLR-11278: Stopping CDCR should cancel a running bootstrap operation. (Amrit Sarkar, shalin)
-* SOLR-11293: Potential data loss in TLOG replicas after replication failures (noble)
+* SOLR-11293: Potential data loss in TLOG replicas when masterVersion equals zero (noble, Cao Manh Dat)
* SOLR-10101: TestLazyCores hangs (Erick Erickson)
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/cc9c8670/solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java b/solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java
index 8a6b99b..73e6142 100644
--- a/solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java
+++ b/solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java
@@ -214,6 +214,7 @@ public class RecoveryStrategy implements Runnable, Closeable {
ModifiableSolrParams solrParams = new ModifiableSolrParams();
solrParams.set(ReplicationHandler.MASTER_URL, leaderUrl);
+ solrParams.set(ReplicationHandler.SKIP_COMMIT_ON_MASTER_VERSION_ZERO, replicaType == Replica.Type.TLOG);
if (isClosed()) return; // we check closed on return
boolean success = replicationHandler.doFetch(solrParams, false).getSuccessful();
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/cc9c8670/solr/core/src/java/org/apache/solr/cloud/ReplicateFromLeader.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/ReplicateFromLeader.java b/solr/core/src/java/org/apache/solr/cloud/ReplicateFromLeader.java
index 549ef70..0a742e3 100644
--- a/solr/core/src/java/org/apache/solr/cloud/ReplicateFromLeader.java
+++ b/solr/core/src/java/org/apache/solr/cloud/ReplicateFromLeader.java
@@ -26,6 +26,7 @@ import org.apache.solr.common.util.NamedList;
import org.apache.solr.core.CoreContainer;
import org.apache.solr.core.SolrConfig;
import org.apache.solr.core.SolrCore;
+import org.apache.solr.handler.IndexFetcher;
import org.apache.solr.handler.ReplicationHandler;
import org.apache.solr.request.LocalSolrQueryRequest;
import org.apache.solr.request.SolrQueryRequest;
@@ -74,6 +75,7 @@ public class ReplicateFromLeader {
NamedList<Object> slaveConfig = new NamedList<>();
slaveConfig.add("fetchFromLeader", Boolean.TRUE);
+ slaveConfig.add(ReplicationHandler.SKIP_COMMIT_ON_MASTER_VERSION_ZERO, switchTransactionLog);
slaveConfig.add("pollInterval", pollIntervalStr);
NamedList<Object> replicationConfig = new NamedList<>();
replicationConfig.add("slave", slaveConfig);
@@ -85,10 +87,11 @@ public class ReplicateFromLeader {
replicationProcess = new ReplicationHandler();
if (switchTransactionLog) {
- replicationProcess.setPollListener((solrCore, pollSuccess) -> {
- if (pollSuccess) {
+ replicationProcess.setPollListener((solrCore, fetchResult) -> {
+ if (fetchResult == IndexFetcher.IndexFetchResult.INDEX_FETCH_SUCCESS) {
String commitVersion = getCommitVersion(core);
if (commitVersion == null) return;
+ if (Long.parseLong(commitVersion) == lastVersion) return;
UpdateLog updateLog = solrCore.getUpdateHandler().getUpdateLog();
SolrQueryRequest req = new LocalSolrQueryRequest(core,
new ModifiableSolrParams());
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/cc9c8670/solr/core/src/java/org/apache/solr/cloud/ZkController.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/ZkController.java b/solr/core/src/java/org/apache/solr/cloud/ZkController.java
index b3793af..dcce2c8 100644
--- a/solr/core/src/java/org/apache/solr/cloud/ZkController.java
+++ b/solr/core/src/java/org/apache/solr/cloud/ZkController.java
@@ -989,7 +989,7 @@ public class ZkController {
if (isTlogReplicaAndNotLeader) {
String commitVersion = ReplicateFromLeader.getCommitVersion(core);
if (commitVersion != null) {
- ulog.copyOverOldUpdates(Long.parseLong(commitVersion), true);
+ ulog.copyOverOldUpdates(Long.parseLong(commitVersion));
}
}
// we will call register again after zk expiration and on reload
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/cc9c8670/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java b/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
index 1bf452f..52d3b25 100644
--- a/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
+++ b/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
@@ -163,6 +163,8 @@ public class IndexFetcher {
private Integer soTimeout;
+ private boolean skipCommitOnMasterVersionZero;
+
private static final String INTERRUPT_RESPONSE_MESSAGE = "Interrupted while waiting for modify lock";
public static class IndexFetchResult {
@@ -226,6 +228,10 @@ public class IndexFetcher {
if (fetchFromLeader != null && fetchFromLeader instanceof Boolean) {
this.fetchFromLeader = (boolean) fetchFromLeader;
}
+ Object skipCommitOnMasterVersionZero = initArgs.get(SKIP_COMMIT_ON_MASTER_VERSION_ZERO);
+ if (skipCommitOnMasterVersionZero != null && skipCommitOnMasterVersionZero instanceof Boolean) {
+ this.skipCommitOnMasterVersionZero = (boolean) skipCommitOnMasterVersionZero;
+ }
String masterUrl = (String) initArgs.get(MASTER_URL);
if (masterUrl == null && !this.fetchFromLeader)
throw new SolrException(SolrException.ErrorCode.SERVER_ERROR,
@@ -428,7 +434,7 @@ public class IndexFetcher {
LOG.info("Slave's version: " + IndexDeletionPolicyWrapper.getCommitTimestamp(commit));
if (latestVersion == 0L) {
- if (forceReplication && commit.getGeneration() != 0) {
+ if (commit.getGeneration() != 0) {
// since we won't get the files for an empty index,
// we just clear ours and commit
LOG.info("New index in Master. Deleting mine...");
@@ -438,8 +444,12 @@ public class IndexFetcher {
} finally {
iw.decref();
}
- SolrQueryRequest req = new LocalSolrQueryRequest(solrCore, new ModifiableSolrParams());
- solrCore.getUpdateHandler().commit(new CommitUpdateCommand(req, false));
+ if (skipCommitOnMasterVersionZero) {
+ openNewSearcherAndUpdateCommitPoint();
+ } else {
+ SolrQueryRequest req = new LocalSolrQueryRequest(solrCore, new ModifiableSolrParams());
+ solrCore.getUpdateHandler().commit(new CommitUpdateCommand(req, false));
+ }
}
//there is nothing to be replicated
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/cc9c8670/solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java b/solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java
index 2fef434..a9e14f4 100644
--- a/solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java
@@ -220,7 +220,7 @@ public class ReplicationHandler extends RequestHandlerBase implements SolrCoreAw
private PollListener pollListener;
public interface PollListener {
- void onComplete(SolrCore solrCore, boolean pollSuccess) throws IOException;
+ void onComplete(SolrCore solrCore, IndexFetchResult fetchResult) throws IOException;
}
/**
@@ -1182,8 +1182,8 @@ public class ReplicationHandler extends RequestHandlerBase implements SolrCoreAw
try {
LOG.debug("Polling for index modifications");
markScheduledExecutionStart();
- boolean pollSuccess = doFetch(null, false).getSuccessful();
- if (pollListener != null) pollListener.onComplete(core, pollSuccess);
+ IndexFetchResult fetchResult = doFetch(null, false);
+ if (pollListener != null) pollListener.onComplete(core, fetchResult);
} catch (Exception e) {
LOG.error("Exception in fetching index", e);
}
@@ -1754,6 +1754,10 @@ public class ReplicationHandler extends RequestHandlerBase implements SolrCoreAw
public static final String FETCH_FROM_LEADER = "fetchFromLeader";
+ // in case of TLOG replica, if masterVersion = zero, don't do commit
+ // otherwise updates from current tlog won't copied over properly to the new tlog, leading to data loss
+ public static final String SKIP_COMMIT_ON_MASTER_VERSION_ZERO = "skipCommitOnMasterVersionZero";
+
public static final String STATUS = "status";
public static final String COMMAND = "command";
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/cc9c8670/solr/core/src/java/org/apache/solr/handler/admin/PrepRecoveryOp.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/admin/PrepRecoveryOp.java b/solr/core/src/java/org/apache/solr/handler/admin/PrepRecoveryOp.java
index 748982d..0a6d5ce 100644
--- a/solr/core/src/java/org/apache/solr/handler/admin/PrepRecoveryOp.java
+++ b/solr/core/src/java/org/apache/solr/handler/admin/PrepRecoveryOp.java
@@ -20,7 +20,6 @@ package org.apache.solr.handler.admin;
import java.lang.invoke.MethodHandles;
import java.util.Objects;
-import org.apache.lucene.search.MatchAllDocsQuery;
import org.apache.solr.cloud.CloudDescriptor;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.cloud.ClusterState;
@@ -29,15 +28,10 @@ import org.apache.solr.common.cloud.Replica;
import org.apache.solr.common.cloud.Slice;
import org.apache.solr.common.cloud.ZkStateReader;
import org.apache.solr.common.params.CoreAdminParams;
-import org.apache.solr.common.params.ModifiableSolrParams;
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.core.CoreContainer;
import org.apache.solr.core.SolrCore;
import org.apache.solr.handler.admin.CoreAdminHandler.CallInfo;
-import org.apache.solr.request.LocalSolrQueryRequest;
-import org.apache.solr.search.SolrIndexSearcher;
-import org.apache.solr.update.CommitUpdateCommand;
-import org.apache.solr.util.RefCounted;
import org.apache.solr.util.TestInjection;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -177,33 +171,6 @@ class PrepRecoveryOp implements CoreAdminHandler.CoreAdminOp {
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
"Solr is shutting down");
}
-
- // solrcloud_debug
- if (log.isDebugEnabled() && core != null) {
- try {
- LocalSolrQueryRequest r = new LocalSolrQueryRequest(core,
- new ModifiableSolrParams());
- CommitUpdateCommand commitCmd = new CommitUpdateCommand(r, false);
- commitCmd.softCommit = true;
- core.getUpdateHandler().commit(commitCmd);
- RefCounted<SolrIndexSearcher> searchHolder = core
- .getNewestSearcher(false);
- SolrIndexSearcher searcher = searchHolder.get();
- try {
- log.debug(core.getCoreContainer()
- .getZkController().getNodeName()
- + " to replicate "
- + searcher.search(new MatchAllDocsQuery(), 1).totalHits
- + " gen:"
- + core.getDeletionPolicy().getLatestCommit().getGeneration()
- + " data:" + core.getDataDir());
- } finally {
- searchHolder.decref();
- }
- } catch (Exception e) {
- log.debug("Error in solrcloud_debug block", e);
- }
- }
}
Thread.sleep(1000);
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/cc9c8670/solr/core/src/java/org/apache/solr/update/UpdateLog.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/update/UpdateLog.java b/solr/core/src/java/org/apache/solr/update/UpdateLog.java
index 5a16cd4..fc02917 100644
--- a/solr/core/src/java/org/apache/solr/update/UpdateLog.java
+++ b/solr/core/src/java/org/apache/solr/update/UpdateLog.java
@@ -244,6 +244,7 @@ public class UpdateLog implements PluginInfoInitialized, SolrMetricProducer {
protected Gauge<Integer> bufferedOpsGauge;
protected Meter applyingBufferedOpsMeter;
protected Meter replayOpsMeter;
+ protected Meter copyOverOldUpdatesMeter;
public static class LogPtr {
final long pointer;
@@ -435,6 +436,7 @@ public class UpdateLog implements PluginInfoInitialized, SolrMetricProducer {
manager.registerGauge(null, registry, () -> getTotalLogsSize(), true, "bytes", scope, "replay", "remaining");
applyingBufferedOpsMeter = manager.meter(null, registry, "ops", scope, "applyingBuffered");
replayOpsMeter = manager.meter(null, registry, "ops", scope, "replay");
+ copyOverOldUpdatesMeter = manager.meter(null, registry, "ops", scope, "copyOverOldUpdates");
manager.registerGauge(null, registry, () -> state.getValue(), true, "state", scope);
}
@@ -1158,12 +1160,12 @@ public class UpdateLog implements PluginInfoInitialized, SolrMetricProducer {
protected void copyAndSwitchToNewTlog(CommitUpdateCommand cuc) {
synchronized (this) {
- if (tlog == null && prevTlog == null && prevMapLog2 == null && logs.isEmpty()) {
+ if (tlog == null) {
return;
}
preCommit(cuc);
try {
- copyOverOldUpdates(cuc.getVersion(), false);
+ copyOverOldUpdates(cuc.getVersion());
} finally {
postCommit(cuc);
}
@@ -1173,9 +1175,8 @@ public class UpdateLog implements PluginInfoInitialized, SolrMetricProducer {
/**
* Copy over updates from prevTlog or last tlog (in tlog folder) to a new tlog
* @param commitVersion any updates that have version larger than the commitVersion will be copied over
- * @param omitCommitted if a tlog is already committed then don't read it
*/
- public void copyOverOldUpdates(long commitVersion, boolean omitCommitted) {
+ public void copyOverOldUpdates(long commitVersion) {
TransactionLog oldTlog = prevTlog;
if (oldTlog == null && !logs.isEmpty()) {
oldTlog = logs.getFirst();
@@ -1185,11 +1186,12 @@ public class UpdateLog implements PluginInfoInitialized, SolrMetricProducer {
}
try {
- if (omitCommitted && oldTlog.endsWithCommit()) return;
+ if (oldTlog.endsWithCommit()) return;
} catch (IOException e) {
log.warn("Exception reading log", e);
return;
}
+ copyOverOldUpdatesMeter.mark();
SolrQueryRequest req = new LocalSolrQueryRequest(uhandler.core,
new ModifiableSolrParams());
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/cc9c8670/solr/core/src/test/org/apache/solr/cloud/TestTlogReplica.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestTlogReplica.java b/solr/core/src/test/org/apache/solr/cloud/TestTlogReplica.java
index cc470d6..79dce4d 100644
--- a/solr/core/src/test/org/apache/solr/cloud/TestTlogReplica.java
+++ b/solr/core/src/test/org/apache/solr/cloud/TestTlogReplica.java
@@ -22,6 +22,7 @@ import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.EnumSet;
+import java.util.HashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
@@ -29,6 +30,8 @@ import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
+import com.carrotsearch.randomizedtesting.annotations.Repeat;
+import com.codahale.metrics.Meter;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
@@ -60,8 +63,6 @@ import org.apache.solr.common.util.NamedList;
import org.apache.solr.core.SolrCore;
import org.apache.solr.update.DirectUpdateHandler2;
import org.apache.solr.update.SolrIndexWriter;
-import org.apache.solr.update.UpdateHandler;
-import org.apache.solr.update.UpdateLog;
import org.apache.solr.util.RefCounted;
import org.apache.solr.util.TestInjection;
import org.apache.solr.util.TimeOut;
@@ -71,8 +72,6 @@ import org.junit.BeforeClass;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import com.carrotsearch.randomizedtesting.annotations.Repeat;
-
@Slow
public class TestTlogReplica extends SolrCloudTestCase {
@@ -464,16 +463,13 @@ public class TestTlogReplica extends SolrCloudTestCase {
.process(cloudClient, collectionName);
{
- UpdateHandler updateHandler = getSolrCore(true).get(0).getUpdateHandler();
- RefCounted<IndexWriter> iwRef = updateHandler.getSolrCoreState().getIndexWriter(null);
- assertTrue("IndexWriter at leader must see updates ", iwRef.get().hasUncommittedChanges());
- iwRef.decref();
+ long docsPending = (long) getSolrCore(true).get(0).getMetricRegistry().getGauges().get("UPDATE.updateHandler.docsPending").getValue();
+ assertEquals(4, docsPending);
}
for (SolrCore solrCore : getSolrCore(false)) {
- RefCounted<IndexWriter> iwRef = solrCore.getUpdateHandler().getSolrCoreState().getIndexWriter(null);
- assertFalse("IndexWriter at replicas must not see updates ", iwRef.get().hasUncommittedChanges());
- iwRef.decref();
+ long docsPending = (long) solrCore.getMetricRegistry().getGauges().get("UPDATE.updateHandler.docsPending").getValue();
+ assertEquals(0, docsPending);
}
checkRTG(1, 4, cluster.getJettySolrRunners());
@@ -486,16 +482,12 @@ public class TestTlogReplica extends SolrCloudTestCase {
// The DBQ is not processed at replicas, so we still can get doc2 and other docs by RTG
checkRTG(2,4, getSolrRunner(false));
+ Map<SolrCore, Long> timeCopyOverPerCores = getTimesCopyOverOldUpdates(getSolrCore(false));
new UpdateRequest()
.commit(cloudClient, collectionName);
waitForNumDocsInAllActiveReplicas(2);
-
- // Update log roll over
- for (SolrCore solrCore : getSolrCore(false)) {
- UpdateLog updateLog = solrCore.getUpdateHandler().getUpdateLog();
- assertFalse(updateLog.hasUncommittedChanges());
- }
+ assertCopyOverOldUpdates(1, timeCopyOverPerCores);
// UpdateLog copy over old updates
for (int i = 15; i <= 150; i++) {
@@ -506,6 +498,7 @@ public class TestTlogReplica extends SolrCloudTestCase {
}
checkRTG(120,150, cluster.getJettySolrRunners());
waitForReplicasCatchUp(20);
+ assertCopyOverOldUpdates(2, timeCopyOverPerCores);
}
@SuppressWarnings("unchecked")
@@ -535,7 +528,7 @@ public class TestTlogReplica extends SolrCloudTestCase {
waitForState("Replica didn't recover", collectionName, activeReplicaCount(0,2,0));
// We skip peerSync, so replica will always trigger commit on leader
// We query only the non-leader replicas, since we haven't opened a new searcher on the leader yet
- waitForNumDocsInAllReplicas(4, getNonLeaderReplias(collectionName), 0);// Should be immediate
+ waitForNumDocsInAllReplicas(4, getNonLeaderReplias(collectionName), 10); //timeout for stale collection state
// If I add the doc immediately, the leader fails to communicate with the follower with broken pipe.
// Options are, wait or retry...
@@ -556,8 +549,8 @@ public class TestTlogReplica extends SolrCloudTestCase {
DirectUpdateHandler2.commitOnClose = true;
ChaosMonkey.start(solrRunner);
waitForState("Replica didn't recover", collectionName, activeReplicaCount(0,2,0));
+ waitForNumDocsInAllReplicas(5, getNonLeaderReplias(collectionName), 10); //timeout for stale collection state
checkRTG(3,7, cluster.getJettySolrRunners());
- waitForNumDocsInAllReplicas(5, getNonLeaderReplias(collectionName), 0);// Should be immediate
cluster.getSolrClient().commit(collectionName);
// Test replica recovery apply buffer updates
@@ -910,4 +903,23 @@ public class TestTlogReplica extends SolrCloudTestCase {
fail("Some replicas are not in sync with leader");
}
+
+ private void assertCopyOverOldUpdates(long delta, Map<SolrCore, Long> timesPerCore) {
+ for (SolrCore core : timesPerCore.keySet()) {
+ assertEquals(timesPerCore.get(core) + delta, getTimesCopyOverOldUpdates(core));
+ }
+ }
+
+ private Map<SolrCore, Long> getTimesCopyOverOldUpdates(List<SolrCore> cores) {
+ Map<SolrCore, Long> timesPerCore = new HashMap<>();
+ for (SolrCore core : cores) {
+ long times = getTimesCopyOverOldUpdates(core);
+ timesPerCore.put(core, times);
+ }
+ return timesPerCore;
+ }
+
+ private long getTimesCopyOverOldUpdates(SolrCore core) {
+ return ((Meter)core.getMetricRegistry().getMetrics().get("TLOG.copyOverOldUpdates.ops")).getCount();
+ }
}
[49/50] [abbrv] lucene-solr:feature/autoscaling: Merge branch
'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr into
feature/autoscaling
Posted by no...@apache.org.
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr into feature/autoscaling
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/309ed641
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/309ed641
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/309ed641
Branch: refs/heads/feature/autoscaling
Commit: 309ed641ee7ec464ca36883c1c5fbc9a05501eb5
Parents: 310bcd7
Author: Noble Paul <no...@apache.org>
Authored: Tue Oct 3 23:57:29 2017 +1030
Committer: Noble Paul <no...@apache.org>
Committed: Tue Oct 3 23:57:29 2017 +1030
----------------------------------------------------------------------
solr/CHANGES.txt | 46 +++++++++++++++++++-
.../cloud/OverseerCollectionMessageHandler.java | 5 ++-
.../java/org/apache/solr/cloud/RestoreCmd.java | 2 +-
.../org/apache/solr/cloud/SplitShardCmd.java | 1 +
4 files changed, 50 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/309ed641/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 0383f8b..9451d45 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -73,6 +73,9 @@ Upgrade Notes
makes it much easier for evaluators to handle differing data types (primitives, objects, arrays,
lists, and so forth). (Dennis Gove)
+* SOLR-10962: in the ReplicationHandler the master.commitReserveDuration sub-element is deprecated. Instead
+ please configure a direct commitReserveDuration element for use in all modes (master, slave, cloud).
+
New Features
----------------------
* SOLR-10339: New set-trigger and remove-trigger APIs for autoscaling. (shalin)
@@ -119,6 +122,13 @@ New Features
* SOLR-11316: JSON Facet API: min/max aggregations are now supported on single-valued date fields.
(yonik)
+* SOLR-10962: Make ReplicationHandler's commitReserveDuration configurable in SolrCloud mode.
+ (Ramsey Haddad, Christine Poerschke, hossman)
+
+* SOLR-11382: Lucene's Geo3D (surface of sphere & ellipsoid) is now supported on spatial RPT fields by
+ setting spatialContextFactory="Geo3D". Furthermore, this is the first time Solr has out of the box
+ support for polygons. (David Smiley)
+
Bug Fixes
----------------------
@@ -166,7 +176,7 @@ Bug Fixes
* SOLR-11278: Stopping CDCR should cancel a running bootstrap operation. (Amrit Sarkar, shalin)
-* SOLR-11293: Potential data loss in TLOG replicas after replication failures (noble)
+* SOLR-11293: Potential data loss in TLOG replicas when masterVersion equals zero (noble, Cao Manh Dat)
* SOLR-10101: TestLazyCores hangs (Erick Erickson)
@@ -179,6 +189,17 @@ Bug Fixes
* SOLR-11085: Improve resiliency of autoscaling actions against overseer restarts and operation failures. (shalin)
+* SOLR-11297: Message "Lock held by this virtual machine" during startup. Solr is trying to start some cores twice.
+ (Luiz Armesto, Shawn Heisey, Erick Erickson)
+
+* SOLR-11399: The UnifiedHighlighter was ignoring the hl.fragsize parameter when hl.bs.type=SEPARATOR
+ (Marc Morissette via David Smiley)
+
+* SOLR-11224: SolrStream.close can hit an NPE (Erick Erickson)
+
+* SOLR-11278: Fix a race condition in the CDCR bootstrap process which could lead to bootstraps cancelling itself
+ (Amrit Sarkar, shalin, Varun Thacker)
+
Optimizations
----------------------
@@ -197,6 +218,8 @@ Optimizations
* SOLR-8344: Decide default when requested fields are both column and row stored. (Cao Manh Dat, David Smiley)
+* SOLR-10285: Skip LEADER messages when there are leader only shards (Cao Manh Dat, Joshua Humphries)
+
Other Changes
----------------------
* SOLR-10643: Throttling strategy for triggers and policy executions. (shalin)
@@ -273,6 +296,27 @@ Other Changes
* SOLR-10451: Remove contrib/ltr/lib from lib includes in the techproducts example config (sungjunyoung via janhoy)
+================== 7.0.1 ==================
+
+Consult the LUCENE_CHANGES.txt file for additional, low level, changes in this release.
+
+Versions of Major Components
+---------------------
+Apache Tika 1.13
+Carrot2 3.15.0
+Velocity 1.7 and Velocity Tools 2.0
+Apache UIMA 2.3.1
+Apache ZooKeeper 3.4.10
+Jetty 9.3.14.v20161028
+
+Bug Fixes
+----------------------
+
+* SOLR-11297: Message "Lock held by this virtual machine" during startup. Solr is trying to start some cores twice.
+ (Luiz Armesto, Shawn Heisey, Erick Erickson)
+
+* SOLR-11406: Solr 7.0 cannot read indexes from 6.x versions. (Shawn Heisey, Steve Rowe)
+
================== 7.0.0 ==================
Versions of Major Components
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/309ed641/solr/core/src/java/org/apache/solr/cloud/OverseerCollectionMessageHandler.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/OverseerCollectionMessageHandler.java b/solr/core/src/java/org/apache/solr/cloud/OverseerCollectionMessageHandler.java
index 577cf64..509d17c 100644
--- a/solr/core/src/java/org/apache/solr/cloud/OverseerCollectionMessageHandler.java
+++ b/solr/core/src/java/org/apache/solr/cloud/OverseerCollectionMessageHandler.java
@@ -37,6 +37,7 @@ import org.apache.commons.lang.StringUtils;
import org.apache.solr.client.solrj.SolrResponse;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper;
+import org.apache.solr.common.cloud.DistributedQueue;
import org.apache.solr.client.solrj.impl.HttpSolrClient;
import org.apache.solr.client.solrj.impl.HttpSolrClient.RemoteSolrException;
import org.apache.solr.client.solrj.request.AbstractUpdateRequest;
@@ -321,7 +322,7 @@ public class OverseerCollectionMessageHandler implements OverseerMessageHandler
}
private void processReplicaDeletePropertyCommand(ClusterState clusterState, ZkNodeProps message, NamedList results)
- throws KeeperException, InterruptedException {
+ throws Exception {
checkRequired(message, COLLECTION_PROP, SHARD_ID_PROP, REPLICA_PROP, PROPERTY_PROP);
SolrZkClient zkClient = zkStateReader.getZkClient();
DistributedQueue inQueue = Overseer.getStateUpdateQueue(zkClient);
@@ -332,7 +333,7 @@ public class OverseerCollectionMessageHandler implements OverseerMessageHandler
inQueue.offer(Utils.toJSON(m));
}
- private void balanceProperty(ClusterState clusterState, ZkNodeProps message, NamedList results) throws KeeperException, InterruptedException {
+ private void balanceProperty(ClusterState clusterState, ZkNodeProps message, NamedList results) throws Exception {
if (StringUtils.isBlank(message.getStr(COLLECTION_PROP)) || StringUtils.isBlank(message.getStr(PROPERTY_PROP))) {
throw new SolrException(ErrorCode.BAD_REQUEST,
"The '" + COLLECTION_PROP + "' and '" + PROPERTY_PROP +
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/309ed641/solr/core/src/java/org/apache/solr/cloud/RestoreCmd.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/RestoreCmd.java b/solr/core/src/java/org/apache/solr/cloud/RestoreCmd.java
index 9edf729..f2d3794 100644
--- a/solr/core/src/java/org/apache/solr/cloud/RestoreCmd.java
+++ b/solr/core/src/java/org/apache/solr/cloud/RestoreCmd.java
@@ -32,8 +32,8 @@ import java.util.Objects;
import java.util.Optional;
import java.util.Properties;
import java.util.Set;
-import java.util.concurrent.atomic.AtomicReference;
+import org.apache.solr.common.cloud.DistributedQueue;
import org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper;
import org.apache.solr.cloud.overseer.OverseerAction;
import org.apache.solr.common.cloud.ReplicaPosition;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/309ed641/solr/core/src/java/org/apache/solr/cloud/SplitShardCmd.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/SplitShardCmd.java b/solr/core/src/java/org/apache/solr/cloud/SplitShardCmd.java
index 3bd6d89..bde92d6 100644
--- a/solr/core/src/java/org/apache/solr/cloud/SplitShardCmd.java
+++ b/solr/core/src/java/org/apache/solr/cloud/SplitShardCmd.java
@@ -27,6 +27,7 @@ import java.util.List;
import java.util.Map;
import java.util.Set;
+import org.apache.solr.common.cloud.DistributedQueue;
import org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper;
import org.apache.solr.client.solrj.request.CoreAdminRequest;
import org.apache.solr.cloud.OverseerCollectionMessageHandler.Cmd;
[39/50] [abbrv] lucene-solr:feature/autoscaling: Fix my name spelling
in the CHANGES file
Posted by no...@apache.org.
Fix my name spelling in the CHANGES file
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/472d5202
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/472d5202
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/472d5202
Branch: refs/heads/feature/autoscaling
Commit: 472d52022f00774783a611aed9eba6f6b424ae1b
Parents: 90842fa
Author: Alexandre Rafalovitch <ar...@apache.org>
Authored: Sun Oct 1 18:42:21 2017 -0400
Committer: Alexandre Rafalovitch <ar...@apache.org>
Committed: Sun Oct 1 18:42:21 2017 -0400
----------------------------------------------------------------------
solr/CHANGES.txt | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/472d5202/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index aed0c22..90ada02 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -469,7 +469,7 @@ New Features
* SOLR-10406: v2 API error messages list the URL request path as /solr/____v2/... when the original path was /v2/... (Cao Manh Dat, noble)
* SOLR-10574: New _default config set replacing basic_configs and data_driven_schema_configs.
- (Ishan Chattopadhyaya, noble, shalin, hossman, David Smiley, janhoy, Alexandre Rafalovich)
+ (Ishan Chattopadhyaya, noble, shalin, hossman, David Smiley, janhoy, Alexandre Rafalovitch)
* SOLR-10272: Use _default config set if no collection.configName is specified with CREATE (Ishan Chattopadhyaya)
@@ -477,7 +477,7 @@ New Features
to facilitate both search and faceting. AddSchemaFieldsUpdateProcessor now has the ability to add a "copyField" to
the type mappings, with an optional maxChars limitation. You can also define one typeMappings as default.
This also solves issues SOLR-8495, SOLR-6966, and SOLR-7058
- (janhoy, Steve Rowe, hossman, Alexandre Rafalovich, Shawn Heisey, Cao Manh Dat)
+ (janhoy, Steve Rowe, hossman, Alexandre Rafalovitch, Shawn Heisey, Cao Manh Dat)
* SOLR-10123: Upgraded the Analytics Component to version 2.0 which now supports distributed collections, expressions over
multivalued fields, a new JSON request language, and more. DocValues are now required for any field used in the analytics
[38/50] [abbrv] lucene-solr:feature/autoscaling:
addBackcompatIndexes.py: fix random seed when creating moreterms index
Posted by no...@apache.org.
addBackcompatIndexes.py: fix random seed when creating moreterms index
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/90842fa0
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/90842fa0
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/90842fa0
Branch: refs/heads/feature/autoscaling
Commit: 90842fa0b1bcada723f75420e5f2cc23fdd1f767
Parents: ae5caf3
Author: Steve Rowe <sa...@gmail.com>
Authored: Sat Sep 30 23:37:16 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Sat Sep 30 23:37:16 2017 -0400
----------------------------------------------------------------------
.../test/org/apache/lucene/index/TestBackwardsCompatibility.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/90842fa0/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
index 7437e75..eb6a3e1 100644
--- a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
+++ b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
@@ -152,11 +152,10 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
MockAnalyzer analyzer = new MockAnalyzer(random());
analyzer.setMaxTokenLength(TestUtil.nextInt(random(), 1, IndexWriter.MAX_TERM_LENGTH));
- // TODO: remove randomness
IndexWriterConfig conf = new IndexWriterConfig(analyzer)
.setMergePolicy(mp).setUseCompoundFile(false);
IndexWriter writer = new IndexWriter(dir, conf);
- LineFileDocs docs = new LineFileDocs(random());
+ LineFileDocs docs = new LineFileDocs(new Random(0));
for(int i=0;i<50;i++) {
writer.addDocument(docs.nextDoc());
}
[26/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11165: Fix
broken asciidoc link
Posted by no...@apache.org.
SOLR-11165: Fix broken asciidoc link
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/eaa55867
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/eaa55867
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/eaa55867
Branch: refs/heads/feature/autoscaling
Commit: eaa558672b904b6918a3f5fc3b965b0e39cd0284
Parents: 86fa1cf
Author: Alan Woodward <ro...@apache.org>
Authored: Fri Sep 29 10:05:06 2017 +0100
Committer: Alan Woodward <ro...@apache.org>
Committed: Fri Sep 29 10:05:06 2017 +0100
----------------------------------------------------------------------
solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/eaa55867/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc
index dcfe737..4ea7612 100644
--- a/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc
+++ b/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc
@@ -77,7 +77,7 @@ The above create collection command will associate a policy named `policy1` with
Note that the collection-specific policy is applied *in addition to* the cluster policy, i.e., it is not an override but an augmentation. Therefore the collection will follow all conditions laid out in the cluster preferences, cluster policy, and the policy named `policy1`.
-You can learn more about collection-specific policies in the section <<solrcloud-autoscaling-policy-preferences.adoc#defining-collection-specific-policies,Defining Collection-Specific Policies>>.
+You can learn more about collection-specific policies in the section <<solrcloud-autoscaling-policy-preferences.adoc#collection-specific-policy,Defining Collection-Specific Policies>>.
== Triggers
[02/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11297: Message
'Lock held by this virtual machine' during startup. Solr is trying to start
some cores twice
Posted by no...@apache.org.
SOLR-11297: Message 'Lock held by this virtual machine' during startup. Solr is trying to start some cores twice
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/6391a75a
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/6391a75a
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/6391a75a
Branch: refs/heads/feature/autoscaling
Commit: 6391a75a50ecc05db0d7a5ed9adc9fe187a4f57e
Parents: feec5c6
Author: Erick Erickson <er...@apache.org>
Authored: Mon Sep 25 12:12:31 2017 -0700
Committer: Erick Erickson <er...@apache.org>
Committed: Mon Sep 25 12:12:31 2017 -0700
----------------------------------------------------------------------
solr/CHANGES.txt | 3 ++
.../org/apache/solr/core/CoreContainer.java | 42 +++++++++++++++++---
2 files changed, 40 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/6391a75a/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 0c58639..18219d4 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -145,6 +145,9 @@ Bug Fixes
* SOLR-11363: JSON Facet API: repeated values in a numeric points field with docValues enabled
were double counted. (Hossman, yonik)
+* SOLR-11297: Message "Lock held by this virtual machine" during startup. Solr is trying to start some cores twice.
+ (Luiz Armesto, Shawn Heisey, Erick Erickson)
+
Optimizations
----------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/6391a75a/solr/core/src/java/org/apache/solr/core/CoreContainer.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/core/CoreContainer.java b/solr/core/src/java/org/apache/solr/core/CoreContainer.java
index bf24db8..f3f3c49 100644
--- a/solr/core/src/java/org/apache/solr/core/CoreContainer.java
+++ b/solr/core/src/java/org/apache/solr/core/CoreContainer.java
@@ -637,9 +637,10 @@ public class CoreContainer {
if (zkSys.getZkController() != null) {
zkSys.getZkController().throwErrorIfReplicaReplaced(cd);
}
-
+ solrCores.waitAddPendingCoreOps(cd.getName());
core = createFromDescriptor(cd, false, false);
} finally {
+ solrCores.removeFromPendingOps(cd.getName());
if (asyncSolrCoreLoad) {
solrCores.markCoreAsNotLoading(cd);
}
@@ -935,7 +936,13 @@ public class CoreContainer {
// first and clean it up if there's an error.
coresLocator.create(this, cd);
- SolrCore core = createFromDescriptor(cd, true, newCollection);
+ SolrCore core = null;
+ try {
+ solrCores.waitAddPendingCoreOps(cd.getName());
+ core = createFromDescriptor(cd, true, newCollection);
+ } finally {
+ solrCores.removeFromPendingOps(cd.getName());
+ }
return core;
} catch (Exception ex) {
@@ -970,7 +977,6 @@ public class CoreContainer {
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
"Error CREATEing SolrCore '" + coreName + "': " + ex.getMessage() + rootMsg, ex);
}
-
}
/**
@@ -979,6 +985,26 @@ public class CoreContainer {
* @param dcore a core descriptor
* @param publishState publish core state to the cluster if true
*
+ * WARNING: Any call to this method should be surrounded by a try/finally block
+ * that calls solrCores.waitAddPendingCoreOps(...) and solrCores.removeFromPendingOps(...)
+ *
+ * <pre>
+ * <code>
+ * try {
+ * solrCores.waitAddPendingCoreOps(dcore.getName());
+ * createFromDescriptor(...);
+ * } finally {
+ * solrCores.removeFromPendingOps(dcore.getName());
+ * }
+ * </code>
+ * </pre>
+ *
+ * Trying to put the waitAddPending... in this method results in Bad Things Happening due to race conditions.
+ * getCore() depends on getting the core returned _if_ it's in the pending list due to some other thread opening it.
+ * If the core is not in the pending list and not loaded, then getCore() calls this method. Anything that called
+ * to check if the core was loaded _or_ in pending ops and, based on the return called createFromDescriptor would
+ * introduce a race condition, see getCore() for the place it would be a problem
+ *
* @return the newly created core
*/
private SolrCore createFromDescriptor(CoreDescriptor dcore, boolean publishState, boolean newCollection) {
@@ -1258,7 +1284,12 @@ public class CoreContainer {
} else {
CoreLoadFailure clf = coreInitFailures.get(name);
if (clf != null) {
- createFromDescriptor(clf.cd, true, false);
+ try {
+ solrCores.waitAddPendingCoreOps(clf.cd.getName());
+ createFromDescriptor(clf.cd, true, false);
+ } finally {
+ solrCores.removeFromPendingOps(clf.cd.getName());
+ }
} else {
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "No such core: " + name );
}
@@ -1431,7 +1462,8 @@ public class CoreContainer {
// TestLazyCores
if (desc == null || zkSys.getZkController() != null) return null;
- // This will put an entry in pending core ops if the core isn't loaded
+ // This will put an entry in pending core ops if the core isn't loaded. Here's where moving the
+ // waitAddPendingCoreOps to createFromDescriptor would introduce a race condition.
core = solrCores.waitAddPendingCoreOps(name);
if (isShutDown) return null; // We're quitting, so stop. This needs to be after the wait above since we may come off
[28/50] [abbrv] lucene-solr:feature/autoscaling: LUCENE-7981 - CTB
should check that CR is not null
Posted by no...@apache.org.
LUCENE-7981 - CTB should check that CR is not null
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/97f246b4
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/97f246b4
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/97f246b4
Branch: refs/heads/feature/autoscaling
Commit: 97f246b4ba3f77f87751a374c587a2c84d3406ab
Parents: f9f6d3a
Author: Tommaso Teofili <to...@apache.org>
Authored: Fri Sep 29 15:34:43 2017 +0200
Committer: Tommaso Teofili <to...@apache.org>
Committed: Fri Sep 29 15:34:52 2017 +0200
----------------------------------------------------------------------
.../org/apache/lucene/classification/ClassificationTestBase.java | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/97f246b4/lucene/classification/src/test/org/apache/lucene/classification/ClassificationTestBase.java
----------------------------------------------------------------------
diff --git a/lucene/classification/src/test/org/apache/lucene/classification/ClassificationTestBase.java b/lucene/classification/src/test/org/apache/lucene/classification/ClassificationTestBase.java
index 6c8f7fd..b3220b0 100644
--- a/lucene/classification/src/test/org/apache/lucene/classification/ClassificationTestBase.java
+++ b/lucene/classification/src/test/org/apache/lucene/classification/ClassificationTestBase.java
@@ -88,6 +88,7 @@ public abstract class ClassificationTestBase<T> extends LuceneTestCase {
protected ClassificationResult<T> checkCorrectClassification(Classifier<T> classifier, String inputDoc, T expectedResult) throws Exception {
ClassificationResult<T> classificationResult = classifier.assignClass(inputDoc);
+ assertNotNull(classificationResult);
T assignedClass = classificationResult.getAssignedClass();
assertNotNull(assignedClass);
assertEquals("got an assigned class of " + assignedClass, expectedResult instanceof BytesRef ? ((BytesRef) expectedResult).utf8ToString() : expectedResult, assignedClass instanceof BytesRef ? ((BytesRef) assignedClass).utf8ToString() : assignedClass);
[23/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11401: Add
zipFDistribution Stream Evaluator
Posted by no...@apache.org.
SOLR-11401: Add zipFDistribution Stream Evaluator
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/790533da
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/790533da
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/790533da
Branch: refs/heads/feature/autoscaling
Commit: 790533da124ab10b52fad35372ed09e5f0bd57d5
Parents: 26677ab
Author: Joel Bernstein <jb...@apache.org>
Authored: Wed Sep 27 21:40:27 2017 -0400
Committer: Joel Bernstein <jb...@apache.org>
Committed: Wed Sep 27 21:47:43 2017 -0400
----------------------------------------------------------------------
.../org/apache/solr/handler/StreamHandler.java | 1 +
.../io/eval/ZipFDistributionEvaluator.java | 48 ++++++++++++++++++++
.../solrj/io/stream/StreamExpressionTest.java | 27 +++++++++++
3 files changed, 76 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/790533da/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
index 56f8bdd..559ccea 100644
--- a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
@@ -305,6 +305,7 @@ public class StreamHandler extends RequestHandlerBase implements SolrCoreAware,
.withFunctionName("mean", MeanEvaluator.class)
.withFunctionName("mode", ModeEvaluator.class)
.withFunctionName("logNormalDistribution", LogNormalDistributionEvaluator.class)
+ .withFunctionName("zipFDistribution", ZipFDistributionEvaluator.class)
// Boolean Stream Evaluators
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/790533da/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/ZipFDistributionEvaluator.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/ZipFDistributionEvaluator.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/ZipFDistributionEvaluator.java
new file mode 100644
index 0000000..e34a8a8
--- /dev/null
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/ZipFDistributionEvaluator.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.io.eval;
+
+import java.io.IOException;
+import java.util.Locale;
+
+import org.apache.commons.math3.distribution.ZipfDistribution;
+import org.apache.solr.client.solrj.io.stream.expr.StreamExpression;
+import org.apache.solr.client.solrj.io.stream.expr.StreamFactory;
+
+public class ZipFDistributionEvaluator extends RecursiveNumericEvaluator implements TwoValueWorker {
+
+ private static final long serialVersionUID = 1;
+
+ public ZipFDistributionEvaluator(StreamExpression expression, StreamFactory factory) throws IOException {
+ super(expression, factory);
+ }
+
+ @Override
+ public Object doWork(Object first, Object second) throws IOException{
+ if(null == first){
+ throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - null found for the first value",toExpression(constructingFactory)));
+ }
+ if(null == second){
+ throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - null found for the second value",toExpression(constructingFactory)));
+ }
+
+ Number size = (Number)first;
+ Number exp = (Number)second;
+
+ return new ZipfDistribution(size.intValue(), exp.doubleValue());
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/790533da/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
index 318f8f6..1326740 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
@@ -6372,6 +6372,33 @@ public class StreamExpressionTest extends SolrCloudTestCase {
}
@Test
+ public void testZipFDistribution() throws Exception {
+ String cexpr = "let(a=sample(zipFDistribution(10, 1), 50000), b=freqTable(a), c=col(b, count))";
+ ModifiableSolrParams paramsLoc = new ModifiableSolrParams();
+ paramsLoc.set("expr", cexpr);
+ paramsLoc.set("qt", "/stream");
+ String url = cluster.getJettySolrRunners().get(0).getBaseUrl().toString()+"/"+COLLECTIONORALIAS;
+ TupleStream solrStream = new SolrStream(url, paramsLoc);
+ StreamContext context = new StreamContext();
+ solrStream.setStreamContext(context);
+ List<Tuple> tuples = getTuples(solrStream);
+ assertTrue(tuples.size() == 1);
+ List<Number> counts = (List<Number>)tuples.get(0).get("c");
+
+ assertTrue(counts.size() == 10);
+
+ int lastCount = Integer.MAX_VALUE;
+ for(Number number : counts) {
+ int current = number.intValue();
+ if(current > lastCount) {
+ throw new Exception("Zipf distribution not descending!!!");
+ } else {
+ lastCount = current;
+ }
+ }
+ }
+
+ @Test
public void testEnumeratedDistribution() throws Exception {
String cexpr = "let(a=uniformIntegerDistribution(1, 10)," +
" b=sample(a, 10000)," +
[10/50] [abbrv] lucene-solr:feature/autoscaling: Merge branch
'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr
Posted by no...@apache.org.
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/ec5704e4
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/ec5704e4
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/ec5704e4
Branch: refs/heads/feature/autoscaling
Commit: ec5704e4af47b971eba8b6870bb007e14e1089de
Parents: 49583ed 7752c05
Author: Karl Wright <Da...@gmail.com>
Authored: Tue Sep 26 03:20:41 2017 -0400
Committer: Karl Wright <Da...@gmail.com>
Committed: Tue Sep 26 03:20:41 2017 -0400
----------------------------------------------------------------------
.../maven/lucene/spatial3d/pom.xml.template | 13 +
lucene/CHANGES.txt | 3 +
.../directory/DirectoryTaxonomyWriter.java | 2 +-
lucene/ivy-versions.properties | 2 +-
.../morfologik-ukrainian-search-3.7.6.jar.sha1 | 1 -
.../morfologik-ukrainian-search-3.9.0.jar.sha1 | 1 +
solr/CHANGES.txt | 3 +
.../org/apache/solr/core/CoreContainer.java | 42 +-
.../org/apache/solr/handler/StreamHandler.java | 6 +-
.../src/basic-authentication-plugin.adoc | 4 +-
.../src/collapse-and-expand-results.adoc | 20 +-
solr/solr-ref-guide/src/enabling-ssl.adoc | 18 +-
solr/solr-ref-guide/src/faceting.adoc | 4 +-
.../src/implicit-requesthandlers.adoc | 19 +-
.../src/major-changes-in-solr-7.adoc | 4 +-
.../src/making-and-restoring-backups.adoc | 5 +-
solr/solr-ref-guide/src/managed-resources.adoc | 2 +-
solr/solr-ref-guide/src/managing-solr.adoc | 2 +-
.../src/mbean-request-handler.adoc | 13 +-
.../src/meta-docs/asciidoc-syntax.adoc | 10 +-
.../src/near-real-time-searching.adoc | 8 +-
.../src/performance-statistics-reference.adoc | 2 +-
solr/solr-ref-guide/src/query-re-ranking.adoc | 2 +-
solr/solr-ref-guide/src/response-writers.adoc | 28 +-
.../src/rule-based-authorization-plugin.adoc | 21 +-
.../src/running-solr-on-hdfs.adoc | 3 +-
...rcloud-query-routing-and-read-tolerance.adoc | 3 +-
.../src/statistical-programming.adoc | 190 ++-
.../src/stream-decorator-reference.adoc | 1216 ++++++++++++++++++
solr/solr-ref-guide/src/stream-decorators.adoc | 1216 ------------------
.../src/stream-evaluator-reference.adoc | 966 ++++++++++++++
solr/solr-ref-guide/src/stream-evaluators.adoc | 1030 ---------------
.../src/stream-source-reference.adoc | 539 ++++++++
solr/solr-ref-guide/src/stream-sources.adoc | 540 --------
.../src/streaming-expressions.adoc | 8 +-
.../src/the-query-elevation-component.adoc | 24 +-
.../solr-ref-guide/src/the-stats-component.adoc | 8 +-
.../src/the-term-vector-component.adoc | 3 +-
.../solr-ref-guide/src/the-terms-component.adoc | 18 +-
solr/solr-ref-guide/src/working-with-dates.adoc | 12 +-
.../client/solrj/io/eval/MeanEvaluator.java | 57 +
.../io/eval/WeibullDistributionEvaluator.java | 48 +
.../solrj/io/stream/StreamExpressionTest.java | 47 +
43 files changed, 3165 insertions(+), 2998 deletions(-)
----------------------------------------------------------------------
[48/50] [abbrv] lucene-solr:feature/autoscaling: Merge branch
'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr into
feature/autoscaling
Posted by no...@apache.org.
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr into feature/autoscaling
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/310bcd7c
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/310bcd7c
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/310bcd7c
Branch: refs/heads/feature/autoscaling
Commit: 310bcd7cbe7cd1563fbd83c85a4c25df7d3eb02f
Parents: 942b671 fd2b4f3
Author: Noble Paul <no...@apache.org>
Authored: Tue Oct 3 23:57:15 2017 +1030
Committer: Noble Paul <no...@apache.org>
Committed: Tue Oct 3 23:57:15 2017 +1030
----------------------------------------------------------------------
.gitignore | 1 +
README.md | 8 +
dev-tools/doap/lucene.rdf | 7 +
dev-tools/doap/solr.rdf | 7 +
.../lucene/backward-codecs/pom.xml.template | 13 +
.../lucene/spatial-extras/pom.xml.template | 7 +
.../maven/lucene/spatial3d/pom.xml.template | 13 +
dev-tools/maven/pom.xml.template | 5 +
.../maven/solr/core/src/test/pom.xml.template | 7 +
dev-tools/scripts/addBackcompatIndexes.py | 69 +-
dev-tools/scripts/buildAndPushRelease.py | 11 +-
lucene/CHANGES.txt | 22 +
.../index/TestBackwardsCompatibility.java | 49 +-
.../org/apache/lucene/index/dvupdates.7.0.0.zip | Bin 0 -> 13401 bytes
.../org/apache/lucene/index/empty.7.0.0.zip | Bin 0 -> 375 bytes
.../org/apache/lucene/index/index.7.0.0-cfs.zip | Bin 0 -> 15594 bytes
.../apache/lucene/index/index.7.0.0-nocfs.zip | Bin 0 -> 15586 bytes
.../org/apache/lucene/index/moreterms.7.0.0.zip | Bin 0 -> 175822 bytes
.../org/apache/lucene/index/sorted.7.0.0.zip | Bin 0 -> 74558 bytes
.../classification/ClassificationTestBase.java | 1 +
.../java/org/apache/lucene/util/Version.java | 7 +
.../lucene/search/TestIntRangeFieldQueries.java | 2 +-
.../directory/DirectoryTaxonomyWriter.java | 2 +-
lucene/ivy-versions.properties | 8 +-
.../morfologik-ukrainian-search-3.7.6.jar.sha1 | 1 -
.../morfologik-ukrainian-search-3.9.0.jar.sha1 | 1 +
.../apache/lucene/queries/mlt/MoreLikeThis.java | 16 +-
.../document/FloatPointNearestNeighbor.java | 383 ++++++
.../lucene/document/LatLonBoundingBox.java | 231 ++++
.../org/apache/lucene/document/LatLonPoint.java | 3 +-
.../document/TestFloatPointNearestNeighbor.java | 239 ++++
.../search/TestLatLonBoundingBoxQueries.java | 299 +++++
lucene/site/xsl/index.xsl | 2 +-
lucene/spatial-extras/build.xml | 9 +
.../spatial/spatial4j/Geo3dBinaryCodec.java | 130 ++
.../spatial/spatial4j/Geo3dCircleShape.java | 82 ++
.../spatial4j/Geo3dDistanceCalculator.java | 154 +++
.../spatial/spatial4j/Geo3dPointShape.java | 78 ++
.../spatial/spatial4j/Geo3dRectangleShape.java | 163 +++
.../lucene/spatial/spatial4j/Geo3dShape.java | 164 ++-
.../spatial/spatial4j/Geo3dShapeFactory.java | 396 ++++++
.../spatial4j/Geo3dSpatialContextFactory.java | 94 ++
.../apache/lucene/spatial/SpatialArgsTest.java | 12 +-
.../lucene/spatial/spatial4j/Geo3dRptTest.java | 193 +--
.../Geo3dShapeRectRelationTestCase.java | 20 +-
.../Geo3dShapeSphereModelRectRelationTest.java | 29 +-
.../Geo3dShapeWGS84ModelRectRelationTest.java | 21 +
.../apache/lucene/spatial3d/geom/Bounded.java | 35 +
.../lucene/spatial3d/geom/GeoCircleFactory.java | 16 +
.../lucene/spatial3d/geom/GeoExactCircle.java | 373 ++++++
.../apache/lucene/spatial3d/geom/GeoShape.java | 10 +-
.../lucene/spatial3d/geom/PlanetModel.java | 61 +-
.../lucene/spatial3d/geom/StandardObjects.java | 1 +
.../lucene/spatial3d/geom/GeoCircleTest.java | 83 ++
.../geom/RandomGeo3dShapeGenerator.java | 40 +-
.../java/org/apache/lucene/geo/GeoTestUtil.java | 5 +
.../search/BaseRangeFieldQueryTestCase.java | 10 +-
solr/common-build.xml | 9 +-
solr/core/build.xml | 3 +-
.../org/apache/solr/cloud/DeleteShardCmd.java | 1 +
.../org/apache/solr/cloud/DistributedQueue.java | 481 -------
.../org/apache/solr/cloud/ElectionContext.java | 38 +-
.../java/org/apache/solr/cloud/Overseer.java | 28 +-
.../apache/solr/cloud/OverseerTaskQueue.java | 4 +-
.../org/apache/solr/cloud/RecoveryStrategy.java | 1 +
.../apache/solr/cloud/ReplicateFromLeader.java | 7 +-
.../org/apache/solr/cloud/ZkController.java | 6 +-
.../apache/solr/cloud/ZkDistributedQueue.java | 489 +++++++
.../org/apache/solr/core/CoreContainer.java | 42 +-
.../apache/solr/handler/CdcrRequestHandler.java | 9 +-
.../org/apache/solr/handler/IndexFetcher.java | 16 +-
.../apache/solr/handler/ReplicationHandler.java | 31 +-
.../org/apache/solr/handler/StreamHandler.java | 15 +-
.../solr/handler/admin/PrepRecoveryOp.java | 33 -
.../solr/highlight/UnifiedSolrHighlighter.java | 18 +-
.../solr/index/SlowCompositeReaderWrapper.java | 16 +-
.../apache/solr/metrics/SolrMetricManager.java | 2 +-
.../solr/schema/AbstractSpatialFieldType.java | 5 +
.../apache/solr/schema/ExternalFileField.java | 1 -
.../java/org/apache/solr/update/UpdateLog.java | 12 +-
.../ClassificationUpdateProcessorParams.java | 2 +-
.../processor/DistributedUpdateProcessor.java | 2 +-
.../processor/UpdateRequestProcessor.java | 1 +
.../solr/collection1/conf/schema-spatial.xml | 4 +
.../collection1/conf/solrconfig-spatial.xml | 6 +
.../solr/configsets/backcompat/conf/schema.xml | 22 +
.../configsets/backcompat/conf/solrconfig.xml | 43 +
.../backcompat/TestLuceneIndexBackCompat.java | 96 ++
.../org/apache/solr/cloud/DeleteShardTest.java | 4 +-
.../apache/solr/cloud/DistributedQueueTest.java | 15 +-
.../org/apache/solr/cloud/ForceLeaderTest.java | 4 +-
.../apache/solr/cloud/MultiThreadedOCPTest.java | 5 +-
.../org/apache/solr/cloud/OverseerTest.java | 22 +-
.../cloud/TestCloudJSONFacetJoinDomain.java | 51 +-
.../cloud/TestRandomRequestDistribution.java | 2 +-
.../solr/cloud/TestShardsWithSingleReplica.java | 84 ++
.../org/apache/solr/cloud/TestTlogReplica.java | 50 +-
.../highlight/TestUnifiedSolrHighlighter.java | 3 +
.../solr/schema/SpatialRPTFieldTypeTest.java | 12 +-
.../apache/solr/search/TestSolr4Spatial2.java | 21 +-
.../asciidoctor-ant-1.6.0-alpha.3.jar.sha1 | 1 -
.../asciidoctor-ant-1.6.0-alpha.5.jar.sha1 | 1 +
solr/site/quickstart.mdtext | 614 +--------
.../src/basic-authentication-plugin.adoc | 4 +-
solr/solr-ref-guide/src/blob-store-api.adoc | 5 +-
.../src/collapse-and-expand-results.adoc | 20 +-
solr/solr-ref-guide/src/enabling-ssl.adoc | 18 +-
solr/solr-ref-guide/src/errata.adoc | 4 +-
solr/solr-ref-guide/src/faceting.adoc | 4 +-
.../solr-tutorial/tutorial-add-copy-field.png | Bin 123194 -> 93190 bytes
.../images/solr-tutorial/tutorial-add-field.png | Bin 212142 -> 110529 bytes
.../src/implicit-requesthandlers.adoc | 19 +-
solr/solr-ref-guide/src/index-replication.adoc | 4 +-
solr/solr-ref-guide/src/installing-solr.adoc | 2 +-
.../src/major-changes-in-solr-7.adoc | 12 +-
.../src/making-and-restoring-backups.adoc | 5 +-
solr/solr-ref-guide/src/managed-resources.adoc | 2 +-
solr/solr-ref-guide/src/managing-solr.adoc | 2 +-
.../src/mbean-request-handler.adoc | 13 +-
.../src/meta-docs/asciidoc-syntax.adoc | 10 +-
solr/solr-ref-guide/src/metrics-reporting.adoc | 21 +-
.../src/near-real-time-searching.adoc | 8 +-
.../src/pdf/SolrRefGuide-all.adoc | 1 +
.../src/performance-statistics-reference.adoc | 160 ++-
solr/solr-ref-guide/src/query-re-ranking.adoc | 2 +-
solr/solr-ref-guide/src/response-writers.adoc | 28 +-
.../src/rule-based-authorization-plugin.adoc | 23 +-
.../src/running-solr-on-hdfs.adoc | 3 +-
solr/solr-ref-guide/src/solr-tutorial.adoc | 10 +-
.../src/solrcloud-autoscaling-actions.adoc | 28 +
.../src/solrcloud-autoscaling-api.adoc | 4 +-
...solrcloud-autoscaling-auto-add-replicas.adoc | 21 +
.../solrcloud-autoscaling-fault-tolerance.adoc | 60 +
.../src/solrcloud-autoscaling-listeners.adoc | 179 +++
.../src/solrcloud-autoscaling-overview.adoc | 43 +-
.../src/solrcloud-autoscaling-triggers.adoc | 106 ++
.../src/solrcloud-autoscaling.adoc | 7 +-
...rcloud-query-routing-and-read-tolerance.adoc | 6 +-
solr/solr-ref-guide/src/spatial-search.adoc | 71 +-
.../src/statistical-programming.adoc | 190 ++-
.../src/stream-decorator-reference.adoc | 1216 ++++++++++++++++++
solr/solr-ref-guide/src/stream-decorators.adoc | 1216 ------------------
.../src/stream-evaluator-reference.adoc | 966 ++++++++++++++
solr/solr-ref-guide/src/stream-evaluators.adoc | 1030 ---------------
.../src/stream-source-reference.adoc | 539 ++++++++
solr/solr-ref-guide/src/stream-sources.adoc | 540 --------
.../src/streaming-expressions.adoc | 9 +-
.../src/the-extended-dismax-query-parser.adoc | 3 +-
.../src/the-query-elevation-component.adoc | 24 +-
.../solr-ref-guide/src/the-stats-component.adoc | 8 +-
.../src/the-term-vector-component.adoc | 7 +-
.../solr-ref-guide/src/the-terms-component.adoc | 18 +-
.../src/updating-parts-of-documents.adoc | 6 +-
.../src/upgrading-a-solr-cluster.adoc | 18 +-
...g-data-with-solr-cell-using-apache-tika.adoc | 2 -
.../solr-ref-guide/src/using-jmx-with-solr.adoc | 70 +-
solr/solr-ref-guide/src/v2-api.adoc | 5 -
solr/solr-ref-guide/src/working-with-dates.adoc | 14 +-
.../io/eval/BetaDistributionEvaluator.java | 48 +
.../io/eval/ConstantDistributionEvaluator.java | 44 +
.../eval/EnumeratedDistributionEvaluator.java | 19 +-
.../solrj/io/eval/FactorialEvaluator.java | 54 +
.../io/eval/GammaDistributionEvaluator.java | 48 +
.../io/eval/LogNormalDistributionEvaluator.java | 48 +
.../client/solrj/io/eval/MeanEvaluator.java | 57 +
.../client/solrj/io/eval/ModeEvaluator.java | 64 +
.../solrj/io/eval/MonteCarloEvaluator.java | 76 ++
.../solrj/io/eval/MovingMedianEvaluator.java | 68 +
.../client/solrj/io/eval/SampleEvaluator.java | 30 +-
.../io/eval/WeibullDistributionEvaluator.java | 48 +
.../io/eval/ZipFDistributionEvaluator.java | 48 +
.../solr/client/solrj/io/stream/LetStream.java | 27 +-
.../solr/client/solrj/io/stream/SolrStream.java | 4 +-
.../solr/client/solrj/io/stream/SqlStream.java | 7 +-
.../io/stream/expr/StreamExpressionParser.java | 29 +
.../solrj/io/stream/expr/StreamFactory.java | 6 +
.../solr/common/cloud/DistributedQueue.java | 41 +
.../solrj/io/stream/StreamExpressionTest.java | 324 ++++-
178 files changed, 9053 insertions(+), 4748 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/310bcd7c/solr/core/src/java/org/apache/solr/cloud/Overseer.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/310bcd7c/solr/core/src/java/org/apache/solr/cloud/ZkController.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/310bcd7c/solr/core/src/java/org/apache/solr/core/CoreContainer.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/310bcd7c/solr/core/src/test/org/apache/solr/cloud/OverseerTest.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/310bcd7c/solr/solr-ref-guide/src/solrcloud-autoscaling-api.adoc
----------------------------------------------------------------------
[35/50] [abbrv] lucene-solr:feature/autoscaling:
addBackcompatIndexes.py: Create 'sorted' indexes;
on major releases, produce moreterms, dvupdates, and empty indexes. Add 7.0.0
indexes.
Posted by no...@apache.org.
addBackcompatIndexes.py: Create 'sorted' indexes; on major releases, produce moreterms, dvupdates, and empty indexes. Add 7.0.0 indexes.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/5afd41de
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/5afd41de
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/5afd41de
Branch: refs/heads/feature/autoscaling
Commit: 5afd41de114ff62a755634c4a49091f88bde22cd
Parents: dc47c6e
Author: Steve Rowe <sa...@gmail.com>
Authored: Sat Sep 30 20:44:01 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Sat Sep 30 20:44:01 2017 -0400
----------------------------------------------------------------------
dev-tools/scripts/addBackcompatIndexes.py | 67 ++++++++++++++-----
.../index/TestBackwardsCompatibility.java | 38 ++++++++---
.../org/apache/lucene/index/dvupdates.7.0.0.zip | Bin 0 -> 13401 bytes
.../org/apache/lucene/index/empty.7.0.0.zip | Bin 0 -> 375 bytes
.../org/apache/lucene/index/index.7.0.0-cfs.zip | Bin 0 -> 15594 bytes
.../apache/lucene/index/index.7.0.0-nocfs.zip | Bin 0 -> 15586 bytes
.../org/apache/lucene/index/moreterms.7.0.0.zip | Bin 0 -> 175822 bytes
.../org/apache/lucene/index/sorted.7.0.0.zip | Bin 0 -> 74558 bytes
8 files changed, 79 insertions(+), 26 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/dev-tools/scripts/addBackcompatIndexes.py
----------------------------------------------------------------------
diff --git a/dev-tools/scripts/addBackcompatIndexes.py b/dev-tools/scripts/addBackcompatIndexes.py
index 543fad6..1cada01 100644
--- a/dev-tools/scripts/addBackcompatIndexes.py
+++ b/dev-tools/scripts/addBackcompatIndexes.py
@@ -31,12 +31,24 @@ import re
import shutil
def create_and_add_index(source, indextype, index_version, current_version, temp_dir):
+ if not current_version.is_back_compat_with(index_version):
+ prefix = 'unsupported'
+ else:
+ prefix = {
+ 'cfs': 'index',
+ 'nocfs': 'index',
+ 'sorted': 'sorted',
+ 'moreterms': 'moreterms',
+ 'dvupdates': 'dvupdates',
+ 'emptyIndex': 'empty'
+ }[indextype]
if indextype in ('cfs', 'nocfs'):
dirname = 'index.%s' % indextype
+ filename = '%s.%s-%s.zip' % (prefix, index_version, indextype)
else:
dirname = indextype
- prefix = 'index' if current_version.is_back_compat_with(index_version) else 'unsupported'
- filename = '%s.%s-%s.zip' % (prefix, index_version, indextype)
+ filename = '%s.%s.zip' % (prefix, index_version)
+
print(' creating %s...' % filename, end='', flush=True)
module = 'backward-codecs'
index_dir = os.path.join('lucene', module, 'src/test/org/apache/lucene/index')
@@ -47,7 +59,11 @@ def create_and_add_index(source, indextype, index_version, current_version, temp
test = {
'cfs': 'testCreateCFS',
- 'nocfs': 'testCreateNoCFS'
+ 'nocfs': 'testCreateNoCFS',
+ 'sorted': 'testCreateSortedIndex',
+ 'moreterms': 'testCreateMoreTermsIndex',
+ 'dvupdates': 'testCreateIndexWithDocValuesUpdates',
+ 'emptyIndex': 'testCreateEmptyIndex'
}[indextype]
ant_args = ' '.join([
'-Dtests.bwcdir=%s' % temp_dir,
@@ -83,16 +99,22 @@ def create_and_add_index(source, indextype, index_version, current_version, temp
print('done')
def update_backcompat_tests(types, index_version, current_version):
- print(' adding new indexes to backcompat tests...', end='', flush=True)
+ print(' adding new indexes %s to backcompat tests...' % types, end='', flush=True)
module = 'lucene/backward-codecs'
filename = '%s/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java' % module
- matcher = re.compile(r'final static String\[\] oldNames = {|};' if current_version.is_back_compat_with(index_version)
- else r'final String\[\] unsupportedNames = {|};')
+ if not current_version.is_back_compat_with(index_version):
+ matcher = re.compile(r'final String\[\] unsupportedNames = {|};'),
+ elif 'sorted' in types:
+ matcher = re.compile(r'final static String\[\] oldSortedNames = {|};')
+ else:
+ matcher = re.compile(r'final static String\[\] oldNames = {|};')
+
+ strip_dash_suffix_re = re.compile(r'-.*')
def find_version(x):
x = x.strip()
- end = x.index("-")
- return scriptutil.Version.parse(x[1:end])
+ x = re.sub(strip_dash_suffix_re, '', x) # remove the -suffix if any
+ return scriptutil.Version.parse(x)
class Edit(object):
start = None
@@ -100,10 +122,12 @@ def update_backcompat_tests(types, index_version, current_version):
if self.start:
# find where this version should exist
i = len(buffer) - 1
- v = find_version(buffer[i])
- while i >= self.start and v.on_or_after(index_version):
- i -= 1
+ previous_version_exists = '};' not in line # Version list closure => there are no versions
+ if previous_version_exists: # Only look if there is a version here
v = find_version(buffer[i])
+ while i >= self.start and v.on_or_after(index_version):
+ i -= 1
+ v = find_version(buffer[i])
i += 1 # readjust since we skipped past by 1
# unfortunately python doesn't have a range remove from list...
@@ -111,14 +135,20 @@ def update_backcompat_tests(types, index_version, current_version):
while i < len(buffer) and index_version.on_or_after(find_version(buffer[i])):
buffer.pop(i)
- if i == len(buffer) and not buffer[-1].strip().endswith(","):
+ if i == len(buffer) and previous_version_exists and not buffer[-1].strip().endswith(","):
# add comma
buffer[-1] = buffer[-1].rstrip() + ",\n"
- last = buffer[-1]
- spaces = ' ' * (len(last) - len(last.lstrip()))
+ if previous_version_exists:
+ last = buffer[-1]
+ spaces = ' ' * (len(last) - len(last.lstrip()))
+ else:
+ spaces = ' '
for (j, t) in enumerate(types):
- newline = spaces + ('"%s-%s"' % (index_version, t))
+ if t == 'sorted':
+ newline = spaces + ('"sorted.%s"') % index_version
+ else:
+ newline = spaces + ('"%s-%s"' % (index_version, t))
if j < len(types) - 1 or i < len(buffer):
newline += ','
buffer.insert(i, newline + '\n')
@@ -215,9 +245,16 @@ def main():
current_version = scriptutil.Version.parse(scriptutil.find_current_version())
create_and_add_index(source, 'cfs', c.version, current_version, c.temp_dir)
create_and_add_index(source, 'nocfs', c.version, current_version, c.temp_dir)
+ create_and_add_index(source, 'sorted', c.version, current_version, c.temp_dir)
+ if c.version.minor == 0 and c.version.bugfix == 0:
+ create_and_add_index(source, 'moreterms', c.version, current_version, c.temp_dir)
+ create_and_add_index(source, 'dvupdates', c.version, current_version, c.temp_dir)
+ create_and_add_index(source, 'emptyIndex', c.version, current_version, c.temp_dir)
+ print ('\nMANUAL UPDATE REQUIRED: edit TestBackwardsCompatibility to enable moreterms, dvupdates, and empty index testing')
print('\nAdding backwards compatibility tests')
update_backcompat_tests(['cfs', 'nocfs'], c.version, current_version)
+ update_backcompat_tests(['sorted'], c.version, current_version)
print('\nTesting changes')
check_backcompat_tests()
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
index c202973..88900d4 100644
--- a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
+++ b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
@@ -156,7 +156,7 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
IndexWriterConfig conf = new IndexWriterConfig(analyzer)
.setMergePolicy(mp).setUseCompoundFile(false);
IndexWriter writer = new IndexWriter(dir, conf);
- LineFileDocs docs = new LineFileDocs(null);
+ LineFileDocs docs = new LineFileDocs(random());
for(int i=0;i<50;i++) {
writer.addDocument(docs.nextDoc());
}
@@ -275,12 +275,33 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
dir.close();
}
+ public void testCreateEmptyIndex() throws Exception {
+ Path indexDir = getIndexDir().resolve("emptyIndex");
+ Files.deleteIfExists(indexDir);
+ IndexWriterConfig conf = new IndexWriterConfig(new MockAnalyzer(random()))
+ .setUseCompoundFile(false).setMergePolicy(NoMergePolicy.INSTANCE);
+ try (Directory dir = newFSDirectory(indexDir);
+ IndexWriter writer = new IndexWriter(dir, conf)) {
+ writer.flush();
+ }
+ }
+
final static String[] oldNames = {
+ "7.0.0-cfs",
+ "7.0.0-nocfs"
};
public static String[] getOldNames() {
return oldNames;
}
+
+ final static String[] oldSortedNames = {
+ "sorted.7.0.0"
+ };
+
+ public static String[] getOldSortedNames() {
+ return oldSortedNames;
+ }
final String[] unsupportedNames = {
"1.9.0-cfs",
@@ -765,7 +786,7 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
Directory targetDir2 = newDirectory();
IndexWriter w = new IndexWriter(targetDir2, newIndexWriterConfig(new MockAnalyzer(random())));
IllegalArgumentException e = expectThrows(IllegalArgumentException.class, () -> TestUtil.addIndexesSlowly(w, reader));
- assertEquals(e.getMessage(), "Cannot merge a segment that has been created with major version 6 into this index which has been created by major version 7");
+ assertEquals(e.getMessage(), "Cannot merge a segment that has been created with major version 7 into this index which has been created by major version 8");
w.close();
targetDir2.close();
@@ -1449,14 +1470,13 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
public static final String emptyIndex = "empty.7.0.0.zip";
public void testUpgradeEmptyOldIndex() throws Exception {
- assumeTrue("Reenable when 7.0 is released", false);
Path oldIndexDir = createTempDir("emptyIndex");
TestUtil.unzip(getDataInputStream(emptyIndex), oldIndexDir);
Directory dir = newFSDirectory(oldIndexDir);
newIndexUpgrader(dir).upgrade();
- checkAllSegmentsUpgraded(dir, 6);
+ checkAllSegmentsUpgraded(dir, 7);
dir.close();
}
@@ -1464,7 +1484,6 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
public static final String moreTermsIndex = "moreterms.7.0.0.zip";
public void testMoreTerms() throws Exception {
- assumeTrue("Reenable when 7.0 is released", false);
Path oldIndexDir = createTempDir("moreterms");
TestUtil.unzip(getDataInputStream(moreTermsIndex), oldIndexDir);
Directory dir = newFSDirectory(oldIndexDir);
@@ -1509,7 +1528,6 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
}
public void testDocValuesUpdates() throws Exception {
- assumeTrue("Reenable when 7.0 is released", false);
Path oldIndexDir = createTempDir("dvupdates");
TestUtil.unzip(getDataInputStream(dvUpdatesIndex), oldIndexDir);
Directory dir = newFSDirectory(oldIndexDir);
@@ -1572,12 +1590,10 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
}
public void testSortedIndex() throws Exception {
- assumeTrue("Reenable when 7.0 is released", false);
- String[] versions = new String[] {};
- for(String version : versions) {
+ for(String name : oldSortedNames) {
Path path = createTempDir("sorted");
- InputStream resource = TestBackwardsCompatibility.class.getResourceAsStream("sorted." + version + ".zip");
- assertNotNull("Sorted index index " + version + " not found", resource);
+ InputStream resource = TestBackwardsCompatibility.class.getResourceAsStream(name + ".zip");
+ assertNotNull("Sorted index index " + name + " not found", resource);
TestUtil.unzip(resource, path);
// TODO: more tests
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/dvupdates.7.0.0.zip
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/dvupdates.7.0.0.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/dvupdates.7.0.0.zip
new file mode 100644
index 0000000..6c94eae
Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/dvupdates.7.0.0.zip differ
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/empty.7.0.0.zip
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/empty.7.0.0.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/empty.7.0.0.zip
new file mode 100644
index 0000000..d8dac24
Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/empty.7.0.0.zip differ
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-cfs.zip
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-cfs.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-cfs.zip
new file mode 100644
index 0000000..1419931
Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-cfs.zip differ
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-nocfs.zip
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-nocfs.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-nocfs.zip
new file mode 100644
index 0000000..83c3242
Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-nocfs.zip differ
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/moreterms.7.0.0.zip
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/moreterms.7.0.0.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/moreterms.7.0.0.zip
new file mode 100644
index 0000000..5366c09
Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/moreterms.7.0.0.zip differ
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/sorted.7.0.0.zip
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/sorted.7.0.0.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/sorted.7.0.0.zip
new file mode 100644
index 0000000..13948dc
Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/sorted.7.0.0.zip differ
[21/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11400: Fix
precommit
Posted by no...@apache.org.
SOLR-11400: Fix precommit
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/5e840c9a
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/5e840c9a
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/5e840c9a
Branch: refs/heads/feature/autoscaling
Commit: 5e840c9a1c80b5f4116b1a9623dc275bbdb6b535
Parents: 6b3447d
Author: Joel Bernstein <jb...@apache.org>
Authored: Wed Sep 27 14:17:04 2017 -0400
Committer: Joel Bernstein <jb...@apache.org>
Committed: Wed Sep 27 14:17:04 2017 -0400
----------------------------------------------------------------------
solr/core/src/java/org/apache/solr/handler/StreamHandler.java | 1 -
1 file changed, 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5e840c9a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
index 4810759..56f8bdd 100644
--- a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
@@ -29,7 +29,6 @@ import java.util.Iterator;
import java.util.List;
import java.util.Map;
-import org.apache.commons.math3.distribution.LogNormalDistribution;
import org.apache.solr.client.solrj.io.ModelCache;
import org.apache.solr.client.solrj.io.SolrClientCache;
import org.apache.solr.client.solrj.io.Tuple;
[11/50] [abbrv] lucene-solr:feature/autoscaling: Ref Guide: fix
callout number order in rule-based auth page
Posted by no...@apache.org.
Ref Guide: fix callout number order in rule-based auth page
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/6fe7829d
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/6fe7829d
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/6fe7829d
Branch: refs/heads/feature/autoscaling
Commit: 6fe7829d3e99ce1070b4d6d6b298ad3f0520f802
Parents: ec5704e
Author: Cassandra Targett <ct...@apache.org>
Authored: Tue Sep 26 09:27:21 2017 -0500
Committer: Cassandra Targett <ct...@apache.org>
Committed: Tue Sep 26 09:27:48 2017 -0500
----------------------------------------------------------------------
solr/solr-ref-guide/src/rule-based-authorization-plugin.adoc | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/6fe7829d/solr/solr-ref-guide/src/rule-based-authorization-plugin.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/rule-based-authorization-plugin.adoc b/solr/solr-ref-guide/src/rule-based-authorization-plugin.adoc
index f2cb27d..7b530cb 100644
--- a/solr/solr-ref-guide/src/rule-based-authorization-plugin.adoc
+++ b/solr/solr-ref-guide/src/rule-based-authorization-plugin.adoc
@@ -54,10 +54,10 @@ This example `security.json` shows how the <<basic-authentication-plugin.adoc#ba
There are several things defined in this example:
-<1> Basic authentication and rule-based authorization plugins are enabled.
+<1> Basic authentication plugin is enabled.
<2> All requests w/o credentials will be rejected with a 401 error. Set `'blockUnknown'` to false (or remove it altogether) if you wish to let unauthenticated requests to go through. However, if a particular resource is protected by a rule, they are rejected anyway with a 401 error.
-* A user called 'solr', with a password has been defined.
-<4> Basic authentication and rule-based authorization plugins are enabled.
+<3> A user named 'solr', with a password has been defined.
+<4> Rule-based authorization plugin is enabled.
<5> The 'admin' role has been defined, and it has permission to edit security settings.
<6> The 'solr' user has been defined to the 'admin' role.
[04/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11398: Add
weibullDistribution Stream Evaluator
Posted by no...@apache.org.
SOLR-11398: Add weibullDistribution Stream Evaluator
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/0e5c3aa3
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/0e5c3aa3
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/0e5c3aa3
Branch: refs/heads/feature/autoscaling
Commit: 0e5c3aa3dc5a094d974716b8ee018f7469a8534a
Parents: 001fa28
Author: Joel Bernstein <jb...@apache.org>
Authored: Mon Sep 25 21:34:45 2017 -0400
Committer: Joel Bernstein <jb...@apache.org>
Committed: Mon Sep 25 21:41:55 2017 -0400
----------------------------------------------------------------------
.../org/apache/solr/handler/StreamHandler.java | 6 +--
.../client/solrj/io/eval/MeanEvaluator.java | 57 ++++++++++++++++++++
.../io/eval/WeibullDistributionEvaluator.java | 48 +++++++++++++++++
.../solrj/io/stream/StreamExpressionTest.java | 47 ++++++++++++++++
4 files changed, 155 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/0e5c3aa3/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
index 0f059e3..4f50ca9 100644
--- a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
@@ -301,12 +301,12 @@ public class StreamHandler extends RequestHandlerBase implements SolrCoreAware,
.withFunctionName("movingMedian", MovingMedianEvaluator.class)
.withFunctionName("monteCarlo", MonteCarloEvaluator.class)
.withFunctionName("constantDistribution", ConstantDistributionEvaluator.class)
+ .withFunctionName("weibullDistribution", WeibullDistributionEvaluator.class)
+ .withFunctionName("mean", MeanEvaluator.class)
// Boolean Stream Evaluators
-
-
- .withFunctionName("and", AndEvaluator.class)
+ .withFunctionName("and", AndEvaluator.class)
.withFunctionName("eor", ExclusiveOrEvaluator.class)
.withFunctionName("eq", EqualToEvaluator.class)
.withFunctionName("gt", GreaterThanEvaluator.class)
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/0e5c3aa3/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/MeanEvaluator.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/MeanEvaluator.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/MeanEvaluator.java
new file mode 100644
index 0000000..c2e11f7
--- /dev/null
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/MeanEvaluator.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.io.eval;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Locale;
+
+import org.apache.solr.client.solrj.io.stream.expr.StreamExpression;
+import org.apache.solr.client.solrj.io.stream.expr.StreamFactory;
+import org.apache.commons.math3.stat.StatUtils;
+
+
+public class MeanEvaluator extends RecursiveObjectEvaluator implements OneValueWorker {
+ protected static final long serialVersionUID = 1L;
+
+ public MeanEvaluator(StreamExpression expression, StreamFactory factory) throws IOException{
+ super(expression, factory);
+
+ if(1 != containedEvaluators.size()){
+ throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - expecting exactly 1 value but found %d",expression,containedEvaluators.size()));
+ }
+ }
+
+ @Override
+ public Object doWork(Object value) throws IOException{
+ if(null == value){
+ throw new IOException(String.format(Locale.ROOT, "Unable to find %s(...) because the value is null", constructingFactory.getFunctionName(getClass())));
+ }
+ else if(value instanceof List){
+ List<Number> c = (List<Number>) value;
+ double[] data = new double[c.size()];
+ for(int i=0; i< c.size(); i++) {
+ data[i] = c.get(i).doubleValue();
+ }
+
+ return StatUtils.mean(data);
+ }
+ else{
+ throw new IOException(String.format(Locale.ROOT, "Unable to find %s(...) because the value is not a collection, instead a %s was found", constructingFactory.getFunctionName(getClass()), value.getClass().getSimpleName()));
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/0e5c3aa3/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/WeibullDistributionEvaluator.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/WeibullDistributionEvaluator.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/WeibullDistributionEvaluator.java
new file mode 100644
index 0000000..e1a3463
--- /dev/null
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/WeibullDistributionEvaluator.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.io.eval;
+
+import java.io.IOException;
+import java.util.Locale;
+
+import org.apache.commons.math3.distribution.WeibullDistribution;
+import org.apache.solr.client.solrj.io.stream.expr.StreamExpression;
+import org.apache.solr.client.solrj.io.stream.expr.StreamFactory;
+
+public class WeibullDistributionEvaluator extends RecursiveNumericEvaluator implements TwoValueWorker {
+
+ private static final long serialVersionUID = 1;
+
+ public WeibullDistributionEvaluator(StreamExpression expression, StreamFactory factory) throws IOException {
+ super(expression, factory);
+ }
+
+ @Override
+ public Object doWork(Object first, Object second) throws IOException{
+ if(null == first){
+ throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - null found for the first value",toExpression(constructingFactory)));
+ }
+ if(null == second){
+ throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - null found for the second value",toExpression(constructingFactory)));
+ }
+
+ Number shape = (Number)first;
+ Number scale = (Number)second;
+
+ return new WeibullDistribution(shape.doubleValue(), scale.doubleValue());
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/0e5c3aa3/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
index e1075e3..44b8e54 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
@@ -6553,6 +6553,53 @@ public class StreamExpressionTest extends SolrCloudTestCase {
@Test
+ public void testWeibullDistribution() throws Exception {
+ String cexpr = "let(echo=true, " +
+ "a=percentile(sample(weibullDistribution(1, 10),5000), 50), " +
+ "b=percentile(sample(weibullDistribution(1, 50),5000), 50), " +
+ "c=percentile(sample(weibullDistribution(1, 100),5000), 50)," +
+ "d=percentile(sample(weibullDistribution(4, 10),5000), 50)," +
+ "e=percentile(sample(weibullDistribution(8, 10),5000), 50))";
+
+ ModifiableSolrParams paramsLoc = new ModifiableSolrParams();
+ paramsLoc.set("expr", cexpr);
+ paramsLoc.set("qt", "/stream");
+ String url = cluster.getJettySolrRunners().get(0).getBaseUrl().toString()+"/"+COLLECTIONORALIAS;
+ TupleStream solrStream = new SolrStream(url, paramsLoc);
+ StreamContext context = new StreamContext();
+ solrStream.setStreamContext(context);
+ List<Tuple> tuples = getTuples(solrStream);
+ assertTrue(tuples.size() == 1);
+ Number a = (Number)tuples.get(0).get("a");
+ Number b = (Number)tuples.get(0).get("b");
+ Number c = (Number)tuples.get(0).get("c");
+ Number d = (Number)tuples.get(0).get("d");
+ Number e = (Number)tuples.get(0).get("e");
+ assertTrue(a.doubleValue() < b.doubleValue());
+ assertTrue(b.doubleValue() < c.doubleValue());
+ assertTrue(a.doubleValue() < d.doubleValue());
+ assertTrue(d.doubleValue() < e.doubleValue());
+ }
+
+
+ @Test
+ public void testMean() throws Exception {
+ String cexpr = "mean(array(1,2,3,4,5))";
+ ModifiableSolrParams paramsLoc = new ModifiableSolrParams();
+ paramsLoc.set("expr", cexpr);
+ paramsLoc.set("qt", "/stream");
+ String url = cluster.getJettySolrRunners().get(0).getBaseUrl().toString()+"/"+COLLECTIONORALIAS;
+ TupleStream solrStream = new SolrStream(url, paramsLoc);
+ StreamContext context = new StreamContext();
+ solrStream.setStreamContext(context);
+ List<Tuple> tuples = getTuples(solrStream);
+ assertTrue(tuples.size() == 1);
+ Number mean = (Number)tuples.get(0).get("return-value");
+ assertEquals(mean.doubleValue(), 3.0D, 0.0D);
+ }
+
+
+ @Test
public void testScale() throws Exception {
UpdateRequest updateRequest = new UpdateRequest();
[17/50] [abbrv] lucene-solr:feature/autoscaling: LUCENE-7970: Remove
restriction on maximum cutoff angle, since it's really a radius.
Posted by no...@apache.org.
LUCENE-7970: Remove restriction on maximum cutoff angle, since it's really a radius.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/61d1e96b
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/61d1e96b
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/61d1e96b
Branch: refs/heads/feature/autoscaling
Commit: 61d1e96b80240a9111bafcd40ae7b01c0e8fec77
Parents: c05d34d
Author: Karl Wright <Da...@gmail.com>
Authored: Wed Sep 27 04:41:08 2017 -0400
Committer: Karl Wright <Da...@gmail.com>
Committed: Wed Sep 27 04:41:08 2017 -0400
----------------------------------------------------------------------
.../src/java/org/apache/lucene/spatial3d/geom/GeoExactCircle.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/61d1e96b/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoExactCircle.java
----------------------------------------------------------------------
diff --git a/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoExactCircle.java b/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoExactCircle.java
index e64fd6e..f6711b7 100644
--- a/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoExactCircle.java
+++ b/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoExactCircle.java
@@ -61,7 +61,7 @@ class GeoExactCircle extends GeoBaseCircle {
throw new IllegalArgumentException("Latitude out of bounds");
if (lon < -Math.PI || lon > Math.PI)
throw new IllegalArgumentException("Longitude out of bounds");
- if (cutoffAngle < 0.0 || cutoffAngle > Math.PI)
+ if (cutoffAngle < 0.0)
throw new IllegalArgumentException("Cutoff angle out of bounds");
if (cutoffAngle < Vector.MINIMUM_RESOLUTION)
throw new IllegalArgumentException("Cutoff angle cannot be effectively zero");
[50/50] [abbrv] lucene-solr:feature/autoscaling: Merge branch
'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr into
feature/autoscaling
Posted by no...@apache.org.
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr into feature/autoscaling
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/2c873cd8
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/2c873cd8
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/2c873cd8
Branch: refs/heads/feature/autoscaling
Commit: 2c873cd8c6375c6f54a0309cd45e6dfe009bc7c1
Parents: 309ed64
Author: Noble Paul <no...@apache.org>
Authored: Wed Oct 4 00:08:14 2017 +1030
Committer: Noble Paul <no...@apache.org>
Committed: Wed Oct 4 00:08:14 2017 +1030
----------------------------------------------------------------------
.../org/apache/solr/cloud/autoscaling/TriggerEventQueue.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/2c873cd8/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java b/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java
index 99f641c..36a6d5f 100644
--- a/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java
+++ b/solr/core/src/java/org/apache/solr/cloud/autoscaling/TriggerEventQueue.java
@@ -4,8 +4,8 @@ import java.lang.invoke.MethodHandles;
import java.util.Map;
import org.apache.solr.client.solrj.cloud.autoscaling.TriggerEventType;
-import org.apache.solr.cloud.DistributedQueue;
import org.apache.solr.cloud.Overseer;
+import org.apache.solr.cloud.ZkDistributedQueue;
import org.apache.solr.common.cloud.SolrZkClient;
import org.apache.solr.common.cloud.ZkStateReader;
import org.apache.solr.common.util.Utils;
@@ -17,7 +17,7 @@ import org.slf4j.LoggerFactory;
/**
*
*/
-public class TriggerEventQueue extends DistributedQueue {
+public class TriggerEventQueue extends ZkDistributedQueue {
private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
public static final String ENQUEUE_TIME = "_enqueue_time_";
[05/50] [abbrv] lucene-solr:feature/autoscaling: Ref Guide: clean up
typos & other minor issues; last phase of 7.0 pre-release review
Posted by no...@apache.org.
Ref Guide: clean up typos & other minor issues; last phase of 7.0 pre-release review
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/7752c052
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/7752c052
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/7752c052
Branch: refs/heads/feature/autoscaling
Commit: 7752c052529d3cacbb4294a3109a95887b41e139
Parents: 9beafb6
Author: Cassandra Targett <ct...@apache.org>
Authored: Mon Sep 25 20:38:18 2017 -0500
Committer: Cassandra Targett <ct...@apache.org>
Committed: Mon Sep 25 21:24:40 2017 -0500
----------------------------------------------------------------------
.../src/basic-authentication-plugin.adoc | 4 +-
solr/solr-ref-guide/src/enabling-ssl.adoc | 18 +-
.../src/implicit-requesthandlers.adoc | 19 +-
.../src/making-and-restoring-backups.adoc | 5 +-
solr/solr-ref-guide/src/managed-resources.adoc | 2 +-
solr/solr-ref-guide/src/managing-solr.adoc | 2 +-
.../src/mbean-request-handler.adoc | 13 +-
.../src/meta-docs/asciidoc-syntax.adoc | 2 +-
.../src/performance-statistics-reference.adoc | 2 +-
.../src/rule-based-authorization-plugin.adoc | 21 +-
.../src/running-solr-on-hdfs.adoc | 3 +-
...rcloud-query-routing-and-read-tolerance.adoc | 3 +-
.../src/statistical-programming.adoc | 190 +++++++++----------
13 files changed, 139 insertions(+), 145 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/7752c052/solr/solr-ref-guide/src/basic-authentication-plugin.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/basic-authentication-plugin.adoc b/solr/solr-ref-guide/src/basic-authentication-plugin.adoc
index 2d196cf..b936940 100644
--- a/solr/solr-ref-guide/src/basic-authentication-plugin.adoc
+++ b/solr/solr-ref-guide/src/basic-authentication-plugin.adoc
@@ -51,8 +51,8 @@ An example `security.json` showing both sections is shown below to show how thes
There are several things defined in this file:
<1> Basic authentication and rule-based authorization plugins are enabled.
-<2> A user called 'solr', with a password `'SolrRocks'` has been defined.
-<3> The parameter `"blockUnknown":true` means that unauthenticated requests are not allowed to pass through.
+<2> The parameter `"blockUnknown":true` means that unauthenticated requests are not allowed to pass through.
+<3> A user called 'solr', with a password `'SolrRocks'` has been defined.
<4> The 'admin' role has been defined, and it has permission to edit security settings.
<5> The 'solr' user has been defined to the 'admin' role.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/7752c052/solr/solr-ref-guide/src/enabling-ssl.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/enabling-ssl.adoc b/solr/solr-ref-guide/src/enabling-ssl.adoc
index cbd8754..d310378 100644
--- a/solr/solr-ref-guide/src/enabling-ssl.adoc
+++ b/solr/solr-ref-guide/src/enabling-ssl.adoc
@@ -43,9 +43,9 @@ keytool -genkeypair -alias solr-ssl -keyalg RSA -keysize 2048 -keypass secret -s
The above command will create a keystore file named `solr-ssl.keystore.jks` in the current directory.
-=== Convert the Certificate and Key to PEM Format for Use with cURL
+=== Convert the Certificate and Key to PEM Format for Use with curl
-cURL isn't capable of using JKS formatted keystores, so the JKS keystore needs to be converted to PEM format, which cURL understands.
+curl isn't capable of using JKS formatted keystores, so the JKS keystore needs to be converted to PEM format, which curl understands.
First convert the JKS keystore into PKCS12 format using `keytool`:
@@ -63,7 +63,7 @@ Next convert the PKCS12 format keystore, including both the certificate and the
openssl pkcs12 -in solr-ssl.keystore.p12 -out solr-ssl.pem
----
-If you want to use cURL on OS X Yosemite (10.10), you'll need to create a certificate-only version of the PEM format, as follows:
+If you want to use curl on OS X Yosemite (10.10), you'll need to create a certificate-only version of the PEM format, as follows:
[source,bash]
----
@@ -230,16 +230,16 @@ bin\solr.cmd -cloud -s cloud\node2 -z localhost:2181 -p 7574
[IMPORTANT]
====
-cURL on OS X Mavericks (10.9) has degraded SSL support. For more information and workarounds to allow one-way SSL, see http://curl.haxx.se/mail/archive-2013-10/0036.html. cURL on OS X Yosemite (10.10) is improved - 2-way SSL is possible - see http://curl.haxx.se/mail/archive-2014-10/0053.html .
+curl on OS X Mavericks (10.9) has degraded SSL support. For more information and workarounds to allow one-way SSL, see http://curl.haxx.se/mail/archive-2013-10/0036.html. curl on OS X Yosemite (10.10) is improved - 2-way SSL is possible - see http://curl.haxx.se/mail/archive-2014-10/0053.html .
-The cURL commands in the following sections will not work with the system `curl` on OS X Yosemite (10.10). Instead, the certificate supplied with the `-E` param must be in PKCS12 format, and the file supplied with the `--cacert` param must contain only the CA certificate, and no key (see <<Convert the Certificate and Key to PEM Format for Use with cURL,above>> for instructions on creating this file):
+The curl commands in the following sections will not work with the system `curl` on OS X Yosemite (10.10). Instead, the certificate supplied with the `-E` param must be in PKCS12 format, and the file supplied with the `--cacert` param must contain only the CA certificate, and no key (see <<Convert the Certificate and Key to PEM Format for Use with curl,above>> for instructions on creating this file):
[source,bash]
curl -E solr-ssl.keystore.p12:secret --cacert solr-ssl.cacert.pem ...
====
-NOTE: If your operating system does not include cURL, you can download binaries here: http://curl.haxx.se/download.html
+NOTE: If your operating system does not include curl, you can download binaries here: http://curl.haxx.se/download.html
=== Create a SolrCloud Collection using bin/solr
@@ -259,7 +259,7 @@ bin\solr.cmd create -c mycollection -shards 2
The `create` action will pass the `SOLR_SSL_*` properties set in your include file to the SolrJ code used to create the collection.
-=== Retrieve SolrCloud Cluster Status using cURL
+=== Retrieve SolrCloud Cluster Status using curl
To get the resulting cluster status (again, if you have not enabled client authentication, remove the `-E solr-ssl.pem:secret` option):
@@ -315,9 +315,9 @@ cd example/exampledocs
java -Djavax.net.ssl.keyStorePassword=secret -Djavax.net.ssl.keyStore=../../server/etc/solr-ssl.keystore.jks -Djavax.net.ssl.trustStore=../../server/etc/solr-ssl.keystore.jks -Djavax.net.ssl.trustStorePassword=secret -Durl=https://localhost:8984/solr/mycollection/update -jar post.jar *.xml
----
-=== Query Using cURL
+=== Query Using curl
-Use cURL to query the SolrCloud collection created above, from a directory containing the PEM formatted certificate and key created above (e.g. `example/etc/`) - if you have not enabled client authentication (system property `-Djetty.ssl.clientAuth=true)`, then you can remove the `-E solr-ssl.pem:secret` option:
+Use curl to query the SolrCloud collection created above, from a directory containing the PEM formatted certificate and key created above (e.g. `example/etc/`) - if you have not enabled client authentication (system property `-Djetty.ssl.clientAuth=true)`, then you can remove the `-E solr-ssl.pem:secret` option:
[source,bash]
----
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/7752c052/solr/solr-ref-guide/src/implicit-requesthandlers.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/implicit-requesthandlers.adoc b/solr/solr-ref-guide/src/implicit-requesthandlers.adoc
index c10d93b..077ed91 100644
--- a/solr/solr-ref-guide/src/implicit-requesthandlers.adoc
+++ b/solr/solr-ref-guide/src/implicit-requesthandlers.adoc
@@ -20,8 +20,12 @@
Solr ships with many out-of-the-box RequestHandlers, which are called implicit because they are not configured in `solrconfig.xml`.
+These handlers have pre-defined default parameters, known as _paramsets_, which can be modified if necessary.
+
== List of Implicitly Available Endpoints
+// TODO 7.1 - this doesn't look great in the PDF, redesign the presentation
+
// TODO: Change column width to %autowidth.spread when https://github.com/asciidoctor/asciidoctor-pdf/issues/599 is fixed
[cols="15,20,15,50",options="header"]
@@ -57,17 +61,20 @@ Solr ships with many out-of-the-box RequestHandlers, which are called implicit b
== How to View the Configuration
-You can see configuration for all request handlers, including the implicit request handlers, via the <<config-api.adoc#config-api,Config API>>. E.g. for the `gettingstarted` collection:
+You can see configuration for all request handlers, including the implicit request handlers, via the <<config-api.adoc#config-api,Config API>>. For the `gettingstarted` collection:
-`curl http://localhost:8983/solr/gettingstarted/config/requestHandler`
+[source,text]
+curl http://localhost:8983/solr/gettingstarted/config/requestHandler
-To restrict the results to the configuration for a particular request handler, use the `componentName` request param. E.g. to see just the configuration for the `/export` request handler:
+To restrict the results to the configuration for a particular request handler, use the `componentName` request parameter. To see just the configuration for the `/export` request handler:
-`curl "http://localhost:8983/solr/gettingstarted/config/requestHandler?componentName=/export"`
+[source,text]
+curl "http://localhost:8983/solr/gettingstarted/config/requestHandler?componentName=/export"
-To include the expanded paramset in the response, as well as the effective parameters from merging the paramset params with the built-in params, use the `expandParams` request param. E.g. for the `/export` request handler:
+To include the expanded paramset in the response, as well as the effective parameters from merging the paramset parameters with the built-in parameters, use the `expandParams` request param. For the `/export` request handler, you can make a request like this:
-`curl "http://localhost:8983/solr/gettingstarted/config/requestHandler?componentName=/export&expandParams=true"`
+[source,text]
+curl "http://localhost:8983/solr/gettingstarted/config/requestHandler?componentName=/export&expandParams=true"
== How to Edit the Configuration
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/7752c052/solr/solr-ref-guide/src/making-and-restoring-backups.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/making-and-restoring-backups.adoc b/solr/solr-ref-guide/src/making-and-restoring-backups.adoc
index 38da729..1fee263 100644
--- a/solr/solr-ref-guide/src/making-and-restoring-backups.adoc
+++ b/solr/solr-ref-guide/src/making-and-restoring-backups.adoc
@@ -181,10 +181,10 @@ http://localhost:8983/solr/admin/cores?action=LISTSNAPSHOTS&core=techproducts&co
The list snapshot request parameters are:
-core::
+`core`::
The name of the core to whose snapshots we want to list.
-async::
+`async`::
Request ID to track this action which will be processed asynchronously.
=== Delete Snapshot API
@@ -210,7 +210,6 @@ The name of the core whose snapshot we want to delete
`async`::
Request ID to track this action which will be processed asynchronously
-
== Backup/Restore Storage Repositories
Solr provides interfaces to plug different storage systems for backing up and restoring. For example, you can have a Solr cluster running on a local filesystem like EXT3 but you can backup the indexes to a HDFS filesystem or vice versa.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/7752c052/solr/solr-ref-guide/src/managed-resources.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/managed-resources.adoc b/solr/solr-ref-guide/src/managed-resources.adoc
index deb10cc..af11210 100644
--- a/solr/solr-ref-guide/src/managed-resources.adoc
+++ b/solr/solr-ref-guide/src/managed-resources.adoc
@@ -112,7 +112,7 @@ Now, let’s add a new word to the English stop word list using an HTTP PUT:
curl -X PUT -H 'Content-type:application/json' --data-binary '["foo"]' "http://localhost:8983/solr/techproducts/schema/analysis/stopwords/english"
----
-Here we’re using cURL to PUT a JSON list containing a single word “foo” to the managed English stop words set. Solr will return 200 if the request was successful. You can also put multiple words in a single PUT request.
+Here we’re using curl to PUT a JSON list containing a single word “foo” to the managed English stop words set. Solr will return 200 if the request was successful. You can also put multiple words in a single PUT request.
You can test to see if a specific word exists by sending a GET request for that word as a child resource of the set, such as:
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/7752c052/solr/solr-ref-guide/src/managing-solr.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/managing-solr.adoc b/solr/solr-ref-guide/src/managing-solr.adoc
index 6644d62..7be8c8e 100644
--- a/solr/solr-ref-guide/src/managing-solr.adoc
+++ b/solr/solr-ref-guide/src/managing-solr.adoc
@@ -1,7 +1,7 @@
= Managing Solr
:page-shortname: managing-solr
:page-permalink: managing-solr.html
-:page-children: securing-solr, running-solr-on-hdfs, making-and-restoring-backups, configuring-logging, using-jmx-with-solr, mbean-request-handler, performance-statistics-reference, metrics-reporting, v2-api
+:page-children: securing-solr, running-solr-on-hdfs, making-and-restoring-backups, configuring-logging, metrics-reporting, using-jmx-with-solr, mbean-request-handler, performance-statistics-reference, v2-api
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/7752c052/solr/solr-ref-guide/src/mbean-request-handler.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/mbean-request-handler.adoc b/solr/solr-ref-guide/src/mbean-request-handler.adoc
index 65845ee..45c2fa9 100644
--- a/solr/solr-ref-guide/src/mbean-request-handler.adoc
+++ b/solr/solr-ref-guide/src/mbean-request-handler.adoc
@@ -36,6 +36,7 @@ The output format. This operates the same as the <<response-writers.adoc#respons
== MBeanRequestHandler Examples
+// TODO 7.1 - replace with link to tutorial
The following examples assume you are running Solr's `techproducts` example configuration:
[source,bash]
@@ -45,16 +46,20 @@ bin/solr start -e techproducts
To return information about the CACHE category only:
-`\http://localhost:8983/solr/techproducts/admin/mbeans?cat=CACHE`
+[source,text]
+http://localhost:8983/solr/techproducts/admin/mbeans?cat=CACHE
To return information and statistics about the CACHE category only, formatted in XML:
-`\http://localhost:8983/solr/techproducts/admin/mbeans?stats=true&cat=CACHE&wt=xml`
+[source,text]
+http://localhost:8983/solr/techproducts/admin/mbeans?stats=true&cat=CACHE&wt=xml
To return information for everything, and statistics for everything except the `fieldCache`:
-`\http://localhost:8983/solr/techproducts/admin/mbeans?stats=true&f.fieldCache.stats=false`
+[source,text]
+http://localhost:8983/solr/techproducts/admin/mbeans?stats=true&f.fieldCache.stats=false
To return information and statistics for the `fieldCache` only:
-`\http://localhost:8983/solr/techproducts/admin/mbeans?key=fieldCache&stats=true`
+[source,text]
+http://localhost:8983/solr/techproducts/admin/mbeans?key=fieldCache&stats=true
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/7752c052/solr/solr-ref-guide/src/meta-docs/asciidoc-syntax.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/meta-docs/asciidoc-syntax.adoc b/solr/solr-ref-guide/src/meta-docs/asciidoc-syntax.adoc
index 05a7421..dd9180e 100644
--- a/solr/solr-ref-guide/src/meta-docs/asciidoc-syntax.adoc
+++ b/solr/solr-ref-guide/src/meta-docs/asciidoc-syntax.adoc
@@ -205,7 +205,7 @@ For more information, see the section <<schema-api.adoc#modify-the-schema,Modify
More info: http://asciidoctor.org/docs/user-manual/#inter-document-cross-references
-== Lists
+== Ordered and Unordered Lists
AsciiDoc supports three types of lists:
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/7752c052/solr/solr-ref-guide/src/performance-statistics-reference.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/performance-statistics-reference.adoc b/solr/solr-ref-guide/src/performance-statistics-reference.adoc
index 68c7054..a1846d6 100644
--- a/solr/solr-ref-guide/src/performance-statistics-reference.adoc
+++ b/solr/solr-ref-guide/src/performance-statistics-reference.adoc
@@ -79,7 +79,7 @@ Other types of data such as errors and timeouts are also provided. These are ava
The table below shows the metric names and attributes to request:
-[cols="25,75",options="header"]
+[cols="30,70",options="header"]
|===
|Metric name | Description
|`QUERY./select.errors`
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/7752c052/solr/solr-ref-guide/src/rule-based-authorization-plugin.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/rule-based-authorization-plugin.adoc b/solr/solr-ref-guide/src/rule-based-authorization-plugin.adoc
index 3b84dc6..f2cb27d 100644
--- a/solr/solr-ref-guide/src/rule-based-authorization-plugin.adoc
+++ b/solr/solr-ref-guide/src/rule-based-authorization-plugin.adoc
@@ -40,25 +40,26 @@ This example `security.json` shows how the <<basic-authentication-plugin.adoc#ba
----
{
"authentication":{
- "class":"solr.BasicAuthPlugin",
- "blockUnknown": true,
- "credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}
+ "class":"solr.BasicAuthPlugin", <1>
+ "blockUnknown": true, <2>
+ "credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="} <3>
},
"authorization":{
- "class":"solr.RuleBasedAuthorizationPlugin",
+ "class":"solr.RuleBasedAuthorizationPlugin", <4>
"permissions":[{"name":"security-edit",
- "role":"admin"}],
- "user-role":{"solr":"admin"}
+ "role":"admin"}], <5>
+ "user-role":{"solr":"admin"} <6>
}}
----
There are several things defined in this example:
-* Basic authentication and rule-based authorization plugins are enabled.
+<1> Basic authentication and rule-based authorization plugins are enabled.
+<2> All requests w/o credentials will be rejected with a 401 error. Set `'blockUnknown'` to false (or remove it altogether) if you wish to let unauthenticated requests to go through. However, if a particular resource is protected by a rule, they are rejected anyway with a 401 error.
* A user called 'solr', with a password has been defined.
-* All requests w/o credentials will be rejected with a 401 error. Set `'blockUnknown'` to false (or remove it altogether) if you wish to let unauthenticated requests to go through. However, if a particular resource is protected by a rule, they are rejected anyway with a 401 error.
-* The 'admin' role has been defined, and it has permission to edit security settings.
-* The 'solr' user has been defined to the 'admin' role.
+<4> Basic authentication and rule-based authorization plugins are enabled.
+<5> The 'admin' role has been defined, and it has permission to edit security settings.
+<6> The 'solr' user has been defined to the 'admin' role.
== Permission Attributes
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/7752c052/solr/solr-ref-guide/src/running-solr-on-hdfs.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/running-solr-on-hdfs.adoc b/solr/solr-ref-guide/src/running-solr-on-hdfs.adoc
index 6ca5670..bc6a269 100644
--- a/solr/solr-ref-guide/src/running-solr-on-hdfs.adoc
+++ b/solr/solr-ref-guide/src/running-solr-on-hdfs.adoc
@@ -136,7 +136,8 @@ Pass the location of HDFS client configuration files - needed for HDFS HA for ex
Hadoop can be configured to use the Kerberos protocol to verify user identity when trying to access core services like HDFS. If your HDFS directories are protected using Kerberos, then you need to configure Solr's HdfsDirectoryFactory to authenticate using Kerberos in order to read and write to HDFS. To enable Kerberos authentication from Solr, you need to set the following parameters:
-`solr.hdfs.security.kerberos.enabled`:: false |Set to `true` to enable Kerberos authentication. The default is `false`.
+`solr.hdfs.security.kerberos.enabled`::
+Set to `true` to enable Kerberos authentication. The default is `false`.
`solr.hdfs.security.kerberos.keytabfile`::
A keytab file contains pairs of Kerberos principals and encrypted keys which allows for password-less authentication when Solr attempts to authenticate with secure Hadoop.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/7752c052/solr/solr-ref-guide/src/solrcloud-query-routing-and-read-tolerance.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/solrcloud-query-routing-and-read-tolerance.adoc b/solr/solr-ref-guide/src/solrcloud-query-routing-and-read-tolerance.adoc
index 9de8bbe..65f51a1 100644
--- a/solr/solr-ref-guide/src/solrcloud-query-routing-and-read-tolerance.adoc
+++ b/solr/solr-ref-guide/src/solrcloud-query-routing-and-read-tolerance.adoc
@@ -63,8 +63,7 @@ The client can specify '<<distributed-search-with-index-sharding.adoc#distribute
Example response with `partialResults` flag set to 'true':
-*Solr Response with partialResults*
-
+.Solr Response with partialResults
[source,json]
----
{
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/7752c052/solr/solr-ref-guide/src/statistical-programming.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/statistical-programming.adoc b/solr/solr-ref-guide/src/statistical-programming.adoc
index 460fb26..0d461be 100644
--- a/solr/solr-ref-guide/src/statistical-programming.adoc
+++ b/solr/solr-ref-guide/src/statistical-programming.adoc
@@ -18,27 +18,27 @@
// specific language governing permissions and limitations
// under the License.
-The Streaming Expression language includes a powerful statistical programing syntax with many of the
-features of a functional programming language. The syntax includes *variables*, *data structures*
-and a growing set of *mathematical functions*.
+The Streaming Expression language includes a powerful statistical programing syntax with many of the features of a functional programming language.
-Using the statistical programing syntax Solr's powerful *data retrieval*
-capabilities can be combined with in-depth *statistical analysis*.
+The syntax includes variables, data structures and a growing set of mathematical functions.
-The *data retrieval* methods include:
+Using the statistical programing syntax, Solr's powerful data retrieval
+capabilities can be combined with in-depth statistical analysis.
+
+The data retrieval methods include:
* SQL
* time series aggregation
* random sampling
* faceted aggregation
- * KNN searches
- * topic message queues
+ * K-Nearest Neighbor (KNN) searches
+ * `topic` message queues
* MapReduce (parallel relational algebra)
* JDBC calls to outside databases
* Graph Expressions
-Once the data is retrieved, the statistical programming syntax can be used to create *arrays* from the data so it
-can be *manipulated*, *transformed* and *analyzed*.
+Once the data is retrieved, the statistical programming syntax can be used to create arrays from the data so it
+can be manipulated, transformed and analyzed.
The statistical function library includes functions that perform:
@@ -48,7 +48,7 @@ The statistical function library includes functions that perform:
* Moving averages
* Percentiles
* Simple regression and prediction
-* ANOVA
+* Analysis of covariance (ANOVA)
* Histograms
* Convolution
* Euclidean distance
@@ -56,34 +56,30 @@ The statistical function library includes functions that perform:
* Rank transformation
* Normalization transformation
* Sequences
-* Array manipulation functions (creation, copying, length, scaling, reverse etc...)
-
-The statistical function library is backed by *Apache Commons Math*.
+* Array manipulation functions (creation, copying, length, scaling, reverse, etc.)
-This document provides an overview of the how to apply the variables, data structures
-and mathematical functions.
+The statistical function library is backed by https://commons.apache.org/proper/commons-math/[Apache Commons Math library]. A full discussion of many of the math functions available to streaming expressions is available in the section <<stream-evaluator-reference.adoc#stream-evaluator-reference,Stream Evaluator Reference>>.
-== /stream handler
+This document provides an overview of the how to apply the variables, data structures and mathematical functions.
-Like all Streaming Expressions, the statistical functions can be run by Solr's /stream handler.
+NOTE: Like all streaming expressions, the statistical functions are run by Solr's `/stream` handler. For an overview of this handler, see the section <<streaming-expressions.adoc#streaming-expressions,Streaming Expressions>>.
-== Math
+== Math Functions
-Streaming Expressions contain a suite of *mathematical functions* which can be called on
-their own or as part of a larger expression.
+Streaming expressions contain a suite of mathematical functions which can be called on their own or as part of a larger expression.
-Solr's /stream handler evaluates the mathematical expression and returns a result.
+Solr's `/stream` handler evaluates the mathematical expression and returns a result.
-For example sending the following expression to the /stream handler:
+For example, if you send the following expression to the `/stream` handler:
[source,text]
----
add(1, 1)
----
-Returns the following response:
+You get the following response:
-[source,text]
+[source,json]
----
{
"result-set": {
@@ -109,7 +105,7 @@ pow(10, add(1,1))
Returns the following response:
-[source,text]
+[source,json]
----
{
"result-set": {
@@ -126,9 +122,7 @@ Returns the following response:
}
----
-You can also perform math on a stream of Tuples.
-
-For example:
+You can also perform math on a stream of Tuples. For example:
[source,text]
----
@@ -139,7 +133,7 @@ select(search(collection2, q="*:*", fl="price_f", sort="price_f desc", rows="3")
Returns the following response:
-[source, text]
+[source,json]
----
{
"result-set": {
@@ -165,9 +159,13 @@ Returns the following response:
}
----
-== Array (data structure)
+== Data Structures
-The first data structure we'll explore is the *array*.
+Several types of data can be manipulated with the statistical programming syntax. The following sections explore <<Arrays,arrays>>, <<Tuples,tuples>>, and <<Lists,lists>>.
+
+=== Arrays
+
+The first data structure we'll explore is the array.
We can create an array with the `array` function:
@@ -180,7 +178,7 @@ array(1, 2, 3)
Returns the following response:
-[source,text]
+[source,json]
----
{
"result-set": {
@@ -201,7 +199,7 @@ Returns the following response:
}
----
-We can nest arrays within arrays to form a *matrix*:
+We can nest arrays within arrays to form a matrix:
[source,text]
----
@@ -211,7 +209,7 @@ array(array(1, 2, 3),
Returns the following response:
-[source,text]
+[source,json]
----
{
"result-set": {
@@ -239,9 +237,7 @@ Returns the following response:
}
----
-We can manipulate arrays with functions.
-
-For example we can reverse and array with the `rev` function:
+We can manipulate arrays with functions. For example, we can reverse an array with the `rev` function:
[source,text]
----
@@ -250,7 +246,7 @@ rev(array(1, 2, 3))
Returns the following response:
-[source,text]
+[source,json]
----
{
"result-set": {
@@ -271,18 +267,16 @@ Returns the following response:
}
----
-Arrays can also be built and returned by functions.
-
-For example the sequence function:
+Arrays can also be built and returned by functions. For example, the `sequence` function:
[source,text]
----
sequence(5,0,1)
----
-This returns an array of size *5* starting from *0* with a stride of *1*.
+This returns an array of size `5` starting from `0` with a stride of `1`.
-[source,text]
+[source,json]
----
{
"result-set": {
@@ -305,11 +299,7 @@ This returns an array of size *5* starting from *0* with a stride of *1*.
}
----
-We can perform math on an array.
-
-For example we can scale an array with the `scale` function:
-
-Expression:
+We can perform math on an array. For example, we can scale an array with the `scale` function:
[source,text]
----
@@ -318,7 +308,7 @@ scale(10, sequence(5,0,1))
Returns the following response:
-[source,text]
+[source,json]
----
{
"result-set": {
@@ -341,9 +331,7 @@ Returns the following response:
}
----
-We can perform *statistical analysis* on arrays.
-
-For example we can correlate two sequences with the `corr` function:
+We can perform statistical analysis on arrays For example, we can correlate two sequences with the `corr` function:
[source,text]
----
@@ -352,7 +340,7 @@ corr(sequence(5,1,1), sequence(5,10,10))
Returns the following response:
-[source,text]
+[source,json]
----
{
"result-set": {
@@ -370,12 +358,11 @@ Returns the following response:
----
-== Tuple (data structure)
+=== Tuples
-The *tuple* is the next data structure we'll explore.
+The tuple is the next data structure we'll explore.
-The `tuple` function returns a map of name/value pairs. A tuple is a very flexible data structure
-that can hold values that are strings, numerics, arrays and lists of tuples.
+The `tuple` function returns a map of name/value pairs. A tuple is a very flexible data structure that can hold values that are strings, numerics, arrays and lists of tuples.
A tuple can be used to return a complex result from a statistical expression.
@@ -390,7 +377,7 @@ tuple(title="hello world",
Returns the following response:
----
-[source,text]
+[source,json]
----
{
"result-set": {
@@ -419,12 +406,11 @@ Returns the following response:
}
----
-== List (data structure)
+=== Lists
-Next we have the *list* data structure.
+Next we have the list data structure.
-The `list` function is a data structure that wraps Streaming Expressions and emits all the tuples from the wrapped
-expressions as a single concatenated stream.
+The `list` function is a data structure that wraps streaming expressions and emits all the tuples from the wrapped expressions as a single concatenated stream.
Below is an example of a list of tuples:
@@ -436,7 +422,7 @@ list(tuple(id=1, data=array(1, 2, 3)),
Returns the following response:
-[source,text]
+[source,json]
----
{
@@ -467,14 +453,12 @@ Returns the following response:
}
----
-== Let (setting variables)
+== Setting Variables with let
-The `let` function sets *variables* and runs a Streaming Expression that references the variables. The `let` funtion can be used to
+The `let` function sets variables and runs a streaming expression that references the variables. The `let` function can be used to
write small statistical programs.
-A *variable* can be set to the output of any Streaming Expression.
-
-Here is a very simple example:
+A variable can be set to the output of any streaming expression. Here is a very simple example:
[source,text]
----
@@ -483,7 +467,7 @@ let(a=random(collection2, q="*:*", rows="3", fl="price_f"),
tuple(sample1=a, sample2=b))
----
-The `let` expression above is setting variables *a* and *b* to random
+The `let` expression above is setting variables `a` and `b` to random
samples taken from collection2.
The `let` function then executes the `tuple` streaming expression
@@ -491,7 +475,7 @@ which references the two variables.
Here is the output:
-[source,text]
+[source,json]
----
{
"result-set": {
@@ -529,11 +513,11 @@ Here is the output:
}
----
-== Creating arrays with `col` function
+== Creating Arrays with col Function
The `col` function is used to move a column of numbers from a list of tuples into an `array`.
-This is an important function because Streaming Expressions such as `sql`, `random` and `timeseries` return tuples,
-but the statistical functions operate on arrays.
+
+This is an important function because streaming expressions such as `sql`, `random` and `timeseries` return tuples, but the statistical functions operate on arrays.
Below is an example of the `col` function:
@@ -546,20 +530,19 @@ let(a=random(collection2, q="*:*", rows="3", fl="price_f"),
tuple(sample1=c, sample2=d))
----
-The example above is using the `col` function to create arrays from the tuples stored in
-variables *a* and *b*.
+The example above is using the `col` function to create arrays from the tuples stored in variables `a` and `b`.
-Variable *c* contains an array of values from the *price_f* field,
-taken from the tuples stored in variable *a*.
+Variable `c` contains an array of values from the `price_f` field,
+taken from the tuples stored in variable `a`.
-Variable *d* contains an array of values from the *price_f* field,
-taken from the tuples stored in variable *b*.
+Variable `d` contains an array of values from the `price_f` field,
+taken from the tuples stored in variable `b`.
-Also notice inn that the response `tuple` executed by `let` is pointing to the arrays in variables *c* and *d*.
+Also notice inn that the response `tuple` executed by `let` is pointing to the arrays in variables `c` and `d`.
The response shows the arrays:
-[source,text]
+[source,json]
----
{
@@ -588,61 +571,60 @@ The response shows the arrays:
== Statistical Programming Example
-We've covered how the *data structures*, *variables* and a few *statistical functions* work.
-Let's dive into an example that puts these tools to use.
+We've covered how the data structures, variables and a few statistical functions work. Let's dive into an example that puts these tools to use.
-=== Use case
+=== Use Case
We have an existing hotel in *cityA* that is very profitable.
We are contemplating opening up a new hotel in a different city.
We're considering 4 different cities: *cityB*, *cityC*, *cityD*, *cityE*.
-We'd like to open a hotel in a city that has similar room rates to *cityA*.
+We'd like to open a hotel in a city that has similar room rates to cityA.
-How do we determine which of the 4 cities we're considering has room rates which are most similar to *cityA*?
+How do we determine which of the 4 cities we're considering has room rates which are most similar to cityA?
=== The Data
-We have a data set of un-aggregated hotel *bookings*. Each booking record has a rate and city.
+We have a data set of un-aggregated hotel bookings. Each booking record has a rate and city.
-=== Can we simply aggregate?
+=== Can We Simply Aggregate?
-One approach would be to aggregate the data from each city and compare the *mean* room rates. This approach will
+One approach would be to aggregate the data from each city and compare the mean room rates. This approach will
give us some useful information, but the mean is a summary statistic which loses a significant amount of information
-about the data. For example we don't have an understanding of how the distribution of room rates is impacting the
+about the data. For example, we don't have an understanding of how the distribution of room rates is impacting the
mean.
-The *median* room rate provides another interesting data point but it's still not the entire picture. It's sill just
+The median room rate provides another interesting data point but it's still not the entire picture. It's sill just
one point of reference.
Is there a way that we can compare the markets without losing valuable information in the data?
-=== K Nearest Neighbor
+==== K-Nearest Neighbor
-The use case we're reasoning about can often be approached using a K Nearest Neighbor (knn) algorithm.
+The use case we're reasoning about can often be approached using a K-Nearest Neighbor (knn) algorithm.
-With knn we use a *distance* measure to compare vectors of data to find the k nearest neighbors to
+With knn we use a distance measure to compare vectors of data to find the k nearest neighbors to
a specific vector.
-=== Euclidean Distance
+==== Euclidean Distance
-The Streaming Expression statistical function library has a function called `distance`. The `distance` function
+The streaming expression statistical function library has a function called `distance`. The `distance` function
computes the Euclidean distance between two vectors. This looks promising for comparing vectors of room rates.
-=== Vectors
+==== Vectors
But how to create the vectors from a our data set? Remember we have un-aggregated room rates from each of the cities.
How can we vectorize the data so it can be compared using the `distance` function.
-We have a Streaming Expression that can retrieve a *random sample* from each of the cities. The name of this
+We have a streaming expression that can retrieve a random sample from each of the cities. The name of this
expression is `random`. So we could take a random sample of 1000 room rates from each of the five cities.
But random vectors of room rates are not comparable because the distance algorithm compares values at each index
in the vector. How can make these vectors comparable?
-We can make them comparable by *sorting* them. Then as the distance algorithm moves along the vectors it will be
+We can make them comparable by sorting them. Then as the distance algorithm moves along the vectors it will be
comparing room rates from lowest to highest in both cities.
-=== The code
+=== The Code
[source,text]
----
@@ -664,16 +646,16 @@ let(cityA=sort(random(bookings, q="city:cityA", rows="1000", fl="rate_d"), by="r
tuple(city=E, distance=distance(ratesA, ratesE)))))
----
-==== The code explained
+=== The Code Explained
The `let` expression sets variables first.
The first 5 variables (cityA, cityB, cityC, cityD, cityE), contain the random samples from the `bookings` collection.
-the `random` function is pulling 1000 random samples from each city and including the `rate_d` field in the
+The `random` function is pulling 1000 random samples from each city and including the `rate_d` field in the
tuples that are returned.
The `random` function is wrapped by a `sort` function which is sorting the tuples in
-ascending order based on the rate_d field.
+ascending order based on the `rate_d` field.
The next five variables (ratesA, ratesB, ratesC, ratesD, ratesE) contain the arrays of room rates for each
city. The `col` function is used to move the `rate_d` field from the random sample tuples
[08/50] [abbrv] lucene-solr:feature/autoscaling: Ref Guide: rename
Streaming docs to fix broken intra-doc links in the PDF due to mismatching
anchor names
Posted by no...@apache.org.
Ref Guide: rename Streaming docs to fix broken intra-doc links in the PDF due to mismatching anchor names
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/9beafb61
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/9beafb61
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/9beafb61
Branch: refs/heads/feature/autoscaling
Commit: 9beafb612fa22b747b8728d7d954ea6e2bd37844
Parents: 0e5c3aa
Author: Cassandra Targett <ct...@apache.org>
Authored: Mon Sep 25 14:35:52 2017 -0500
Committer: Cassandra Targett <ct...@apache.org>
Committed: Mon Sep 25 21:24:40 2017 -0500
----------------------------------------------------------------------
.../src/major-changes-in-solr-7.adoc | 2 +-
.../src/stream-decorator-reference.adoc | 1216 ++++++++++++++++++
solr/solr-ref-guide/src/stream-decorators.adoc | 1216 ------------------
.../src/stream-evaluator-reference.adoc | 966 ++++++++++++++
solr/solr-ref-guide/src/stream-evaluators.adoc | 966 --------------
.../src/stream-source-reference.adoc | 539 ++++++++
solr/solr-ref-guide/src/stream-sources.adoc | 538 --------
.../src/streaming-expressions.adoc | 8 +-
8 files changed, 2726 insertions(+), 2725 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9beafb61/solr/solr-ref-guide/src/major-changes-in-solr-7.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/major-changes-in-solr-7.adoc b/solr/solr-ref-guide/src/major-changes-in-solr-7.adoc
index 29682df..a82a028 100644
--- a/solr/solr-ref-guide/src/major-changes-in-solr-7.adoc
+++ b/solr/solr-ref-guide/src/major-changes-in-solr-7.adoc
@@ -63,7 +63,7 @@ See the section <<solrcloud-autoscaling.adoc#solrcloud-autoscaling,SolrCloud Aut
** <<highlighting.adoc#the-unified-highlighter,Unified Highlighter>>
** <<metrics-reporting.adoc#metrics-reporting,Metrics API>>. See also information about related deprecations in the section <<JMX Support and MBeans>> below.
** <<other-parsers.adoc#payload-query-parsers,Payload queries>>
-** <<stream-evaluators.adoc#stream-evaluators,Streaming Evaluators>>
+** <<stream-evaluator-reference.adoc#stream-evaluator-reference,Streaming Evaluators>>
** <<v2-api.adoc#v2-api,/v2 API>>
** <<graph-traversal.adoc#graph-traversal,Graph streaming expressions>>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9beafb61/solr/solr-ref-guide/src/stream-decorator-reference.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/stream-decorator-reference.adoc b/solr/solr-ref-guide/src/stream-decorator-reference.adoc
new file mode 100644
index 0000000..5851e45
--- /dev/null
+++ b/solr/solr-ref-guide/src/stream-decorator-reference.adoc
@@ -0,0 +1,1216 @@
+= Stream Decorator Reference
+:page-shortname: stream-decorator-reference
+:page-permalink: stream-decorator-reference.html
+:page-tocclass: right
+:page-toclevels: 1
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+== cartesianProduct
+
+The `cartesianProduct` function turns a single tuple with a multi-valued field (ie. an array) into multiple tuples, one for each value in the array field. That is, given a single tuple containing an array of N values for fieldA, the `cartesianProduct` function will output N tuples, each with one value from the original tuple's array. In essence, you can flatten arrays for further processing.
+
+For example, using `cartesianProduct` you can turn this tuple
+[source,text]
+----
+{
+ "fieldA": "foo",
+ "fieldB": ["bar","baz","bat"]
+}
+----
+
+into the following 3 tuples
+[source,text]
+----
+{
+ "fieldA": "foo",
+ "fieldB": "bar"
+}
+{
+ "fieldA": "foo",
+ "fieldB": "baz"
+}
+{
+ "fieldA": "foo",
+ "fieldB": "bat"
+}
+----
+
+=== cartesianProduct Parameters
+
+* `incoming stream`: (Mandatory) A single incoming stream.
+* `fieldName or evaluator`: (Mandatory) Name of field to flatten values for, or evaluator whose result should be flattened.
+* `productSort='fieldName ASC|DESC'`: (Optional) Sort order of the newly generated tuples.
+
+=== cartesianProduct Syntax
+
+[source,text]
+----
+cartesianProduct(
+ <stream>,
+ <fieldName | evaluator> [as newFieldName],
+ productSort='fieldName ASC|DESC'
+)
+----
+
+=== cartesianProduct Examples
+
+The following examples show different outputs for this source tuple
+
+[source,text]
+----
+{
+ "fieldA": "valueA",
+ "fieldB": ["valueB1","valueB2"],
+ "fieldC": [1,2,3]
+}
+----
+
+==== Single Field, No Sorting
+
+[source,text]
+----
+cartesianProduct(
+ search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
+ fieldB
+)
+
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB1",
+ "fieldC": [1,2,3]
+}
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB2",
+ "fieldC": [1,2,3]
+}
+----
+
+==== Single Evaluator, No Sorting
+
+[source,text]
+----
+cartesianProduct(
+ search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
+ sequence(3,4,5) as fieldE
+)
+
+{
+ "fieldA": "valueA",
+ "fieldB": ["valueB1","valueB2"],
+ "fieldC": [1,2,3],
+ "fieldE": 4
+}
+{
+ "fieldA": "valueA",
+ "fieldB": ["valueB1","valueB2"],
+ "fieldC": [1,2,3],
+ "fieldE": 9
+}
+{
+ "fieldA": "valueA",
+ "fieldB": ["valueB1","valueB2"],
+ "fieldC": [1,2,3],
+ "fieldE": 14
+}
+----
+
+==== Single Field, Sorted by Value
+
+[source,text]
+----
+cartesianProduct(
+ search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
+ fieldB,
+ productSort="fieldB DESC"
+)
+
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB2",
+ "fieldC": [1,2,3]
+}
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB1",
+ "fieldC": [1,2,3]
+}
+----
+
+==== Single Evaluator, Sorted by Evaluator Values
+
+[source,text]
+----
+cartesianProduct(
+ search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
+ sequence(3,4,5) as fieldE,
+ productSort='newFieldE DESC'
+)
+
+{
+ "fieldA": "valueA",
+ "fieldB": ["valueB1","valueB2"],
+ "fieldC": [1,2,3],
+ "fieldE": 14
+}
+{
+ "fieldA": "valueA",
+ "fieldB": ["valueB1","valueB2"],
+ "fieldC": [1,2,3],
+ "fieldE": 9
+}
+{
+ "fieldA": "valueA",
+ "fieldB": ["valueB1","valueB2"],
+ "fieldC": [1,2,3],
+ "fieldE": 4
+}
+----
+
+==== Renamed Single Field, Sorted by Value
+
+[source,text]
+----
+cartesianProduct(
+ search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
+ fieldB as newFieldB,
+ productSort="fieldB DESC"
+)
+
+{
+ "fieldA": "valueA",
+ "fieldB": ["valueB1","valueB2"],
+ "fieldC": [1,2,3]
+ "newFieldB": "valueB2",
+}
+{
+ "fieldA": "valueA",
+ "fieldB": ["valueB1","valueB2"],
+ "fieldC": [1,2,3]
+ "newFieldB": "valueB1",
+}
+----
+
+==== Multiple Fields, No Sorting
+
+[source,text]
+----
+cartesianProduct(
+ search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
+ fieldB,
+ fieldC
+)
+
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB1",
+ "fieldC": 1
+}
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB1",
+ "fieldC": 2
+}
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB1",
+ "fieldC": 3
+}
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB2",
+ "fieldC": 1
+}
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB2",
+ "fieldC": 2
+}
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB2",
+ "fieldC": 3
+}
+----
+
+==== Multiple Fields, Sorted by Single Field
+
+[source,text]
+----
+cartesianProduct(
+ search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
+ fieldB,
+ fieldC,
+ productSort="fieldC ASC"
+)
+
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB1",
+ "fieldC": 1
+}
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB2",
+ "fieldC": 1
+}
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB1",
+ "fieldC": 2
+}
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB2",
+ "fieldC": 2
+}
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB1",
+ "fieldC": 3
+}
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB2",
+ "fieldC": 3
+}
+----
+
+==== Multiple Fields, Sorted by Multiple Fields
+
+[source,text]
+----
+cartesianProduct(
+ search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
+ fieldB,
+ fieldC,
+ productSort="fieldC ASC, fieldB DESC"
+)
+
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB2",
+ "fieldC": 1
+}
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB1",
+ "fieldC": 1
+}
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB2",
+ "fieldC": 2
+}
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB1",
+ "fieldC": 2
+}
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB2",
+ "fieldC": 3
+}
+{
+ "fieldA": "valueA",
+ "fieldB": "valueB1",
+ "fieldC": 3
+}
+----
+
+==== Field and Evaluator, No Sorting
+
+[source,text]
+----
+cartesianProduct(
+ search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
+ sequence(3,4,5) as fieldE,
+ fieldB
+)
+
+{
+ "fieldA": "valueA",
+ "fieldB": valueB1,
+ "fieldC": [1,2,3],
+ "fieldE": 4
+}
+{
+ "fieldA": "valueA",
+ "fieldB": valueB2,
+ "fieldC": [1,2,3],
+ "fieldE": 4
+}
+{
+ "fieldA": "valueA",
+ "fieldB": valueB1,
+ "fieldC": [1,2,3],
+ "fieldE": 9
+}
+{
+ "fieldA": "valueA",
+ "fieldB": valueB2,
+ "fieldC": [1,2,3],
+ "fieldE": 9
+}
+{
+ "fieldA": "valueA",
+ "fieldB": valueB1,
+ "fieldC": [1,2,3],
+ "fieldE": 14
+}
+{
+ "fieldA": "valueA",
+ "fieldB": valueB2,
+ "fieldC": [1,2,3],
+ "fieldE": 14
+}
+----
+
+As you can see in the examples above, the `cartesianProduct` function does support flattening tuples across multiple fields and/or evaluators.
+
+== classify
+
+The `classify` function classifies tuples using a logistic regression text classification model. It was designed specifically to work with models trained using the <<stream-source-reference.adoc#train,train function>>. The `classify` function uses the <<stream-source-reference.adoc#model,model function>> to retrieve a stored model and then scores a stream of tuples using the model. The tuples read by the classifier must contain a text field that can be used for classification. The classify function uses a Lucene analyzer to extract the features from the text so the model can be applied. By default the `classify` function looks for the analyzer using the name of text field in the tuple. If the Solr schema on the worker node does not contain this field, the analyzer can be looked up in another field by specifying the `analyzerField` parameter.
+
+Each tuple that is classified is assigned two scores:
+
+* probability_d* : A float between 0 and 1 which describes the probability that the tuple belongs to the class. This is useful in the classification use case.
+
+* score_d* : The score of the document that has not be squashed between 0 and 1. The score may be positive or negative. The higher the score the better the document fits the class. This un-squashed score will be useful in query re-ranking and recommendation use cases. This score is particularly useful when multiple high ranking documents have a probability_d score of 1, which won't provide a meaningful ranking between documents.
+
+=== classify Parameters
+
+* `model expression`: (Mandatory) Retrieves the stored logistic regression model.
+* `field`: (Mandatory) The field in the tuples to apply the classifier to. By default the analyzer for this field in the schema will be used extract the features.
+* `analyzerField`: (Optional) Specifies a different field to find the analyzer from in the schema.
+
+=== classify Syntax
+
+[source,text]
+----
+classify(model(modelCollection,
+ id="model1",
+ cacheMillis=5000),
+ search(contentCollection,
+ q="id:(a b c)",
+ fl="text_t, id",
+ sort="id asc"),
+ field="text_t")
+----
+
+In the example above the `classify expression` is retrieving the model using the `model` function. It is then classifying tuples returned by the `search` function. The `text_t` field is used for the text classification and the analyzer for the `text_t` field in the Solr schema is used to analyze the text and extract the features.
+
+== commit
+
+The `commit` function wraps a single stream (A) and given a collection and batch size will send commit messages to the collection when the batch size is fulfilled or the end of stream is reached. A commit stream is used most frequently with an update stream and as such the commit will take into account possible summary tuples coming from the update stream. All tuples coming into the commit stream will be returned out of the commit stream - no tuples will be dropped and no tuples will be added.
+
+=== commit Parameters
+
+* `collection`: The collection to send commit messages to (required)
+* `batchSize`: The commit batch size, sends commit message when batch size is hit. If not provided (or provided as value 0) then a commit is only sent at the end of the incoming stream.
+* `waitFlush`: The value passed directly to the commit handler (true/false, default: false)
+* `waitSearcher`: The value passed directly to the commit handler (true/false, default: false)
+* `softCommit`: The value passed directly to the commit handler (true/false, default: false)
+* `StreamExpression for StreamA` (required)
+
+=== commit Syntax
+
+[source,text]
+----
+commit(
+ destinationCollection,
+ batchSize=2,
+ update(
+ destinationCollection,
+ batchSize=5,
+ search(collection1, q=*:*, fl="id,a_s,a_i,a_f,s_multi,i_multi", sort="a_f asc, a_i asc")
+ )
+)
+----
+
+== complement
+
+The `complement` function wraps two streams (A and B) and emits tuples from A which do not exist in B. The tuples are emitted in the order in which they appear in stream A. Both streams must be sorted by the fields being used to determine equality (using the `on` parameter).
+
+=== complement Parameters
+
+* `StreamExpression for StreamA`
+* `StreamExpression for StreamB`
+* `on`: Fields to be used for checking equality of tuples between A and B. Can be of the format `on="fieldName"`, `on="fieldNameInLeft=fieldNameInRight"`, or `on="fieldName, otherFieldName=rightOtherFieldName"`.
+
+=== complement Syntax
+
+[source,text]
+----
+complement(
+ search(collection1, q=a_s:(setA || setAB), fl="id,a_s,a_i", sort="a_i asc, a_s asc"),
+ search(collection1, q=a_s:(setB || setAB), fl="id,a_s,a_i", sort="a_i asc"),
+ on="a_i"
+)
+
+complement(
+ search(collection1, q=a_s:(setA || setAB), fl="id,a_s,a_i", sort="a_i asc, a_s asc"),
+ search(collection1, q=a_s:(setB || setAB), fl="id,a_s,a_i", sort="a_i asc, a_s asc"),
+ on="a_i,a_s"
+)
+----
+
+== daemon
+
+The `daemon` function wraps another function and runs it at intervals using an internal thread. The `daemon` function can be used to provide both continuous push and pull streaming.
+
+=== Continuous Push Streaming
+
+With continuous push streaming the `daemon` function wraps another function and is then sent to the `/stream` handler for execution. The `/stream` handler recognizes the `daemon` function and keeps it resident in memory, so it can run its internal function at intervals.
+
+In order to facilitate the pushing of tuples, the `daemon` function must wrap another stream decorator that pushes the tuples somewhere. One example of this is the `update` function, which wraps a stream and sends the tuples to another SolrCloud collection for indexing.
+
+=== daemon Syntax
+
+[source,text]
+----
+daemon(id="uniqueId",
+ runInterval="1000",
+ terminate="true",
+ update(destinationCollection,
+ batchSize=100,
+ topic(checkpointCollection,
+ topicCollection,
+ q="topic query",
+ fl="id, title, abstract, text",
+ id="topicId",
+ initialCheckpoint=0)
+ )
+ )
+----
+
+The sample code above shows a `daemon` function wrapping an `update` function, which is wrapping a `topic` function. When this expression is sent to the `/stream` handler, the `/stream` hander sees the `daemon` function and keeps it in memory where it will run at intervals. In this particular example, the `daemon` function will run the `update` function every second. The `update` function is wrapping a <<stream-source-reference.adoc#topic,`topic` function>>, which will stream tuples that match the `topic` function query in batches. Each subsequent call to the topic will return the next batch of tuples for the topic. The `update` function will send all the tuples matching the topic to another collection to be indexed. The `terminate` parameter tells the daemon to terminate when the `topic` function stops sending tuples.
+
+The effect of this is to push documents that match a specific query into another collection. Custom push functions can be plugged in that push documents out of Solr and into other systems, such as Kafka or an email system.
+
+Push streaming can also be used for continuous background aggregation scenarios where aggregates are rolled up in the background at intervals and pushed to other Solr collections. Another use case is continuous background machine learning model optimization, where the optimized model is pushed to another Solr collection where it can be integrated into queries.
+
+The `/stream` handler supports a small set commands for listing and controlling daemon functions:
+
+[source,text]
+----
+http://localhost:8983/collection/stream?action=list
+----
+
+This command will provide a listing of the current daemon's running on the specific node along with there current state.
+
+[source,text]
+----
+http://localhost:8983/collection/stream?action=stop&id=daemonId
+----
+
+This command will stop a specific daemon function but leave it resident in memory.
+
+[source,text]
+----
+http://localhost:8983/collection/stream?action=start&id=daemonId
+----
+
+This command will start a specific daemon function that has been stopped.
+
+[source,text]
+----
+http://localhost:8983/collection/stream?action=kill&id=daemonId
+----
+
+This command will stop a specific daemon function and remove it from memory.
+
+=== Continuous Pull Streaming
+
+The {solr-javadocs}/solr-solrj/org/apache/solr/client/solrj/io/stream/DaemonStream.html[DaemonStream] java class (part of the SolrJ libraries) can also be embedded in a java application to provide continuous pull streaming. Sample code:
+
+[source,java]
+----
+StreamContext context = new StreamContext()
+SolrClientCache cache = new SolrClientCache();
+context.setSolrClientCache(cache);
+
+Map topicQueryParams = new HashMap();
+topicQueryParams.put("q","hello"); // The query for the topic
+topicQueryparams.put("rows", "500"); // How many rows to fetch during each run
+topicQueryparams.put("fl", "id", "title"); // The field list to return with the documents
+
+TopicStream topicStream = new TopicStream(zkHost, // Host address for the zookeeper service housing the collections
+ "checkpoints", // The collection to store the topic checkpoints
+ "topicData", // The collection to query for the topic records
+ "topicId", // The id of the topic
+ -1, // checkpoint every X tuples, if set -1 it will checkpoint after each run.
+ topicQueryParams); // The query parameters for the TopicStream
+
+DaemonStream daemonStream = new DaemonStream(topicStream, // The underlying stream to run.
+ "daemonId", // The id of the daemon
+ 1000, // The interval at which to run the internal stream
+ 500); // The internal queue size for the daemon stream. Tuples will be placed in the queue
+ // as they are read by the internal internal thread.
+ // Calling read() on the daemon stream reads records from the internal queue.
+
+daemonStream.setStreamContext(context);
+
+daemonStream.open();
+
+//Read until it's time to shutdown the DaemonStream. You can define the shutdown criteria.
+while(!shutdown()) {
+ Tuple tuple = daemonStream.read() // This will block until tuples become available from the underlying stream (TopicStream)
+ // The EOF tuple (signaling the end of the stream) will never occur until the DaemonStream has been shutdown.
+ //Do something with the tuples
+}
+
+// Shutdown the DaemonStream.
+daemonStream.shutdown();
+
+//Read the DaemonStream until the EOF Tuple is found.
+//This allows the underlying stream to perform an orderly shutdown.
+
+while(true) {
+ Tuple tuple = daemonStream.read();
+ if(tuple.EOF) {
+ break;
+ } else {
+ //Do something with the tuples.
+ }
+}
+//Finally close the stream
+daemonStream.close();
+----
+
+== eval
+
+The `eval` function allows for use cases where new streaming expressions are generated on the fly and then evaluated.
+The `eval` function wraps a streaming expression and reads a single tuple from the underlying stream.
+The `eval` function then retrieves a string Streaming Expressions from the `expr_s` field of the tuple.
+The `eval` function then compiles the string Streaming Expression and emits the tuples.
+
+=== eval Parameters
+
+* `StreamExpression`: (Mandatory) The stream which provides the streaming expression to be evaluated.
+
+=== eval Syntax
+
+[source,text]
+----
+eval(expr)
+----
+
+In the example above the `eval` expression reads the first tuple from the underlying expression. It then compiles and
+executes the string Streaming Expression in the epxr_s field.
+
+== executor
+
+The `executor` function wraps a stream source that contains streaming expressions, and executes the expressions in parallel. The `executor` function looks for the expression in the `expr_s` field in each tuple. The `executor` function has an internal thread pool that runs tasks that compile and run expressions in parallel on the same worker node. This function can also be parallelized across worker nodes by wrapping it in the <<parallel,`parallel`>> function to provide parallel execution of expressions across a cluster.
+
+The `executor` function does not do anything specific with the output of the expressions that it runs. Therefore the expressions that are executed must contain the logic for pushing tuples to their destination. The <<update,update function>> can be included in the expression being executed to send the tuples to a SolrCloud collection for storage.
+
+This model allows for asynchronous execution of jobs where the output is stored in a SolrCloud collection where it can be accessed as the job progresses.
+
+=== executor Parameters
+
+* `threads`: (Optional) The number of threads in the executors thread pool for executing expressions.
+* `StreamExpression`: (Mandatory) The stream source which contains the Streaming Expressions to execute.
+
+=== executor Syntax
+
+[source,text]
+----
+daemon(id="myDaemon",
+ terminate="true",
+ executor(threads=10,
+ topic(checkpointCollection
+ storedExpressions,
+ q="*:*",
+ fl="id, expr_s",
+ initialCheckPoint=0,
+ id="myTopic")))
+----
+
+In the example above a <<daemon,daemon>> wraps an executor, which wraps a <<stream-source-reference.adoc#topic,topic>> that is returning tuples with expressions to execute. When sent to the stream handler, the daemon will call the executor at intervals which will cause the executor to read from the topic and execute the expressions found in the `expr_s` field. The daemon will repeatedly call the executor until all the tuples that match the topic have been iterated, then it will terminate. This is the approach for executing batches of streaming expressions from a `topic` queue.
+
+== fetch
+
+The `fetch` function iterates a stream and fetches additional fields and adds them to the tuples. The `fetch` function fetches in batches to limit the number of calls back to Solr. Tuples streamed from the `fetch` function will contain the original fields and the additional fields that were fetched. The `fetch` function supports one-to-one fetches. Many-to-one fetches, where the stream source contains duplicate keys, will also work, but one-to-many fetches are currently not supported by this function.
+
+=== fetch Parameters
+
+* `Collection`: (Mandatory) The collection to fetch the fields from.
+* `StreamExpression`: (Mandatory) The stream source for the fetch function.
+* `fl`: (Mandatory) The fields to be fetched.
+* `on`: Fields to be used for checking equality of tuples between stream source and fetched records. Formatted as `on="fieldNameInTuple=fieldNameInCollection"`.
+* `batchSize`: (Optional) The batch fetch size.
+
+=== fetch Syntax
+
+[source,text]
+----
+fetch(addresses,
+ search(people, q="*:*", fl="username, firstName, lastName", sort="username asc"),
+ fl="streetAddress, city, state, country, zip",
+ on="username=userId")
+----
+
+The example above fetches addresses for users by matching the username in the tuple with the userId field in the addresses collection.
+
+== having
+
+The `having` expression wraps a stream and applies a boolean operation to each tuple. It emits only tuples for which the boolean operation returns *true*.
+
+=== having Parameters
+
+* `StreamExpression`: (Mandatory) The stream source for the having function.
+* `booleanEvaluator`: (Madatory) The following boolean operations are supported: `eq` (equals), `gt` (greater than), `lt` (less than), `gteq` (greater than or equal to), `lteq` (less than or equal to), `and`, `or`, `eor` (exclusive or), and `not`. Boolean evaluators can be nested with other evaluators to form complex boolean logic.
+
+The comparison evaluators compare the value in a specific field with a value, whether a string, number, or boolean. For example: `eq(field1, 10)`, returns `true` if `field1` is equal to 10.
+
+=== having Syntax
+
+[source,text]
+----
+having(rollup(over=a_s,
+ sum(a_i),
+ search(collection1,
+ q=*:*,
+ fl="id,a_s,a_i,a_f",
+ sort="a_s asc")),
+ and(gt(sum(a_i), 100), lt(sum(a_i), 110)))
+
+----
+
+In this example, the `having` expression iterates the aggregated tuples from the `rollup` expression and emits all tuples where the field `sum(a_i)` is greater then 100 and less then 110.
+
+== leftOuterJoin
+
+The `leftOuterJoin` function wraps two streams, Left and Right, and emits tuples from Left. If there is a tuple in Right equal (as defined by `on`) then the values in that tuple will be included in the emitted tuple. An equal tuple in Right *need not* exist for the Left tuple to be emitted. This supports one-to-one, one-to-many, many-to-one, and many-to-many left outer join scenarios. The tuples are emitted in the order in which they appear in the Left stream. Both streams must be sorted by the fields being used to determine equality (using the `on` parameter). If both tuples contain a field of the same name then the value from the Right stream will be used in the emitted tuple.
+
+You can wrap the incoming streams with a `select` function to be specific about which field values are included in the emitted tuple.
+
+=== leftOuterJoin Parameters
+
+* `StreamExpression for StreamLeft`
+* `StreamExpression for StreamRight`
+* `on`: Fields to be used for checking equality of tuples between Left and Right. Can be of the format `on="fieldName"`, `on="fieldNameInLeft=fieldNameInRight"`, or `on="fieldName, otherFieldName=rightOtherFieldName"`.
+
+=== leftOuterJoin Syntax
+
+[source,text]
+----
+leftOuterJoin(
+ search(people, q=*:*, fl="personId,name", sort="personId asc"),
+ search(pets, q=type:cat, fl="personId,petName", sort="personId asc"),
+ on="personId"
+)
+
+leftOuterJoin(
+ search(people, q=*:*, fl="personId,name", sort="personId asc"),
+ search(pets, q=type:cat, fl="ownerId,petName", sort="ownerId asc"),
+ on="personId=ownerId"
+)
+
+leftOuterJoin(
+ search(people, q=*:*, fl="personId,name", sort="personId asc"),
+ select(
+ search(pets, q=type:cat, fl="ownerId,name", sort="ownerId asc"),
+ ownerId,
+ name as petName
+ ),
+ on="personId=ownerId"
+)
+----
+
+== hashJoin
+
+The `hashJoin` function wraps two streams, Left and Right, and for every tuple in Left which exists in Right will emit a tuple containing the fields of both tuples. This supports one-to-one, one-to-many, many-to-one, and many-to-many inner join scenarios. The tuples are emitted in the order in which they appear in the Left stream. The order of the streams does not matter. If both tuples contain a field of the same name then the value from the Right stream will be used in the emitted tuple.
+
+You can wrap the incoming streams with a `select` function to be specific about which field values are included in the emitted tuple.
+
+The hashJoin function can be used when the tuples of Left and Right cannot be put in the same order. Because the tuples are out of order this stream functions by reading all values from the Right stream during the open operation and will store all tuples in memory. The result of this is a memory footprint equal to the size of the Right stream.
+
+=== hashJoin Parameters
+
+* `StreamExpression for StreamLeft`
+* `hashed=StreamExpression for StreamRight`
+* `on`: Fields to be used for checking equality of tuples between Left and Right. Can be of the format `on="fieldName"`, `on="fieldNameInLeft=fieldNameInRight"`, or `on="fieldName, otherFieldName=rightOtherFieldName"`.
+
+=== hashJoin Syntax
+
+[source,text]
+----
+hashJoin(
+ search(people, q=*:*, fl="personId,name", sort="personId asc"),
+ hashed=search(pets, q=type:cat, fl="personId,petName", sort="personId asc"),
+ on="personId"
+)
+
+hashJoin(
+ search(people, q=*:*, fl="personId,name", sort="personId asc"),
+ hashed=search(pets, q=type:cat, fl="ownerId,petName", sort="ownerId asc"),
+ on="personId=ownerId"
+)
+
+hashJoin(
+ search(people, q=*:*, fl="personId,name", sort="personId asc"),
+ hashed=select(
+ search(pets, q=type:cat, fl="ownerId,name", sort="ownerId asc"),
+ ownerId,
+ name as petName
+ ),
+ on="personId=ownerId"
+)
+----
+
+== innerJoin
+
+Wraps two streams, Left and Right. For every tuple in Left which exists in Right a tuple containing the fields of both tuples will be emitted. This supports one-to-one, one-to-many, many-to-one, and many-to-many inner join scenarios. The tuples are emitted in the order in which they appear in the Left stream. Both streams must be sorted by the fields being used to determine equality (the 'on' parameter). If both tuples contain a field of the same name then the value from the Right stream will be used in the emitted tuple. You can wrap the incoming streams with a `select(...)` expression to be specific about which field values are included in the emitted tuple.
+
+=== innerJoin Parameters
+
+* `StreamExpression for StreamLeft`
+* `StreamExpression for StreamRight`
+* `on`: Fields to be used for checking equality of tuples between Left and Right. Can be of the format `on="fieldName"`, `on="fieldNameInLeft=fieldNameInRight"`, or `on="fieldName, otherFieldName=rightOtherFieldName"`.
+
+=== innerJoin Syntax
+
+[source,text]
+----
+innerJoin(
+ search(people, q=*:*, fl="personId,name", sort="personId asc"),
+ search(pets, q=type:cat, fl="personId,petName", sort="personId asc"),
+ on="personId"
+)
+
+innerJoin(
+ search(people, q=*:*, fl="personId,name", sort="personId asc"),
+ search(pets, q=type:cat, fl="ownerId,petName", sort="ownerId asc"),
+ on="personId=ownerId"
+)
+
+innerJoin(
+ search(people, q=*:*, fl="personId,name", sort="personId asc"),
+ select(
+ search(pets, q=type:cat, fl="ownerId,name", sort="ownerId asc"),
+ ownerId,
+ name as petName
+ ),
+ on="personId=ownerId"
+)
+----
+
+== intersect
+
+The `intersect` function wraps two streams, A and B, and emits tuples from A which *DO* exist in B. The tuples are emitted in the order in which they appear in stream A. Both streams must be sorted by the fields being used to determine equality (the `on` parameter). Only tuples from A are emitted.
+
+=== intersect Parameters
+
+* `StreamExpression for StreamA`
+* `StreamExpression for StreamB`
+* `on`: Fields to be used for checking equality of tuples between A and B. Can be of the format `on="fieldName"`, `on="fieldNameInLeft=fieldNameInRight"`, or `on="fieldName, otherFieldName=rightOtherFieldName"`.
+
+=== intersect Syntax
+
+[source,text]
+----
+intersect(
+ search(collection1, q=a_s:(setA || setAB), fl="id,a_s,a_i", sort="a_i asc, a_s asc"),
+ search(collection1, q=a_s:(setB || setAB), fl="id,a_s,a_i", sort="a_i asc"),
+ on="a_i"
+)
+
+intersect(
+ search(collection1, q=a_s:(setA || setAB), fl="id,a_s,a_i", sort="a_i asc, a_s asc"),
+ search(collection1, q=a_s:(setB || setAB), fl="id,a_s,a_i", sort="a_i asc, a_s asc"),
+ on="a_i,a_s"
+)
+----
+
+== merge
+
+The `merge` function merges two or more streaming expressions and maintains the ordering of the underlying streams. Because the order is maintained, the sorts of the underlying streams must line up with the on parameter provided to the merge function.
+
+=== merge Parameters
+
+* `StreamExpression A`
+* `StreamExpression B`
+* `Optional StreamExpression C,D,....Z`
+* `on`: Sort criteria for performing the merge. Of the form `fieldName order` where order is `asc` or `desc`. Multiple fields can be provided in the form `fieldA order, fieldB order`.
+
+=== merge Syntax
+
+[source,text]
+----
+# Merging two stream expressions together
+merge(
+ search(collection1,
+ q="id:(0 3 4)",
+ fl="id,a_s,a_i,a_f",
+ sort="a_f asc"),
+ search(collection1,
+ q="id:(1)",
+ fl="id,a_s,a_i,a_f",
+ sort="a_f asc"),
+ on="a_f asc")
+----
+
+[source,text]
+----
+# Merging four stream expressions together. Notice that while the sorts of each stream are not identical they are
+# comparable. That is to say the first N fields in each stream's sort matches the N fields in the merge's on clause.
+merge(
+ search(collection1,
+ q="id:(0 3 4)",
+ fl="id,fieldA,fieldB,fieldC",
+ sort="fieldA asc, fieldB desc"),
+ search(collection1,
+ q="id:(1)",
+ fl="id,fieldA",
+ sort="fieldA asc"),
+ search(collection2,
+ q="id:(10 11 13)",
+ fl="id,fieldA,fieldC",
+ sort="fieldA asc"),
+ search(collection3,
+ q="id:(987)",
+ fl="id,fieldA,fieldC",
+ sort="fieldA asc"),
+ on="fieldA asc")
+----
+
+== null
+
+The null expression is a useful utility function for understanding bottlenecks when performing parallel relational algebra (joins, intersections, rollups etc.). The null function reads all the tuples from an underlying stream and returns a single tuple with the count and processing time. Because the null stream adds minimal overhead of it's own, it can be used to isolate the performance of Solr's /export handler. If the /export handlers performance is not the bottleneck, then the bottleneck is likely occurring in the workers where the stream decorators are running.
+
+The null expression can be wrapped by the parallel function and sent to worker nodes. In this scenario each worker will return one tuple with the count of tuples processed on the worker and the timing information for that worker. This gives valuable information such as:
+
+1. As more workers are added does the performance of the /export handler improve or not.
+2. Are tuples being evenly distributed across the workers, or is the hash partitioning sending more documents to a single worker.
+3. Are all workers processing data at the same speed, or is one of the workers the source of the bottleneck.
+
+=== null Parameters
+
+* `StreamExpression`: (Mandatory) The expression read by the null function.
+
+=== null Syntax
+
+[source,text]
+----
+ parallel(workerCollection,
+ null(search(collection1, q=*:*, fl="id,a_s,a_i,a_f", sort="a_s desc", qt="/export", partitionKeys="a_s")),
+ workers="20",
+ zkHost="localhost:9983",
+ sort="a_s desc")
+----
+
+The expression above shows a parallel function wrapping a null function. This will cause the null function to be run in parallel across 20 worker nodes. Each worker will return a single tuple with number of tuples processed and time it took to iterate the tuples.
+
+== outerHashJoin
+
+The `outerHashJoin` function wraps two streams, Left and Right, and emits tuples from Left. If there is a tuple in Right equal (as defined by the `on` parameter) then the values in that tuple will be included in the emitted tuple. An equal tuple in Right *need not* exist for the Left tuple to be emitted. This supports one-to-one, one-to-many, many-to-one, and many-to-many left outer join scenarios. The tuples are emitted in the order in which they appear in the Left stream. The order of the streams does not matter. If both tuples contain a field of the same name then the value from the Right stream will be used in the emitted tuple.
+
+You can wrap the incoming streams with a `select` function to be specific about which field values are included in the emitted tuple.
+
+The outerHashJoin stream can be used when the tuples of Left and Right cannot be put in the same order. Because the tuples are out of order, this stream functions by reading all values from the Right stream during the open operation and will store all tuples in memory. The result of this is a memory footprint equal to the size of the Right stream.
+
+=== outerHashJoin Parameters
+
+* `StreamExpression for StreamLeft`
+* `hashed=StreamExpression for StreamRight`
+* `on`: Fields to be used for checking equality of tuples between Left and Right. Can be of the format `on="fieldName"`, `on="fieldNameInLeft=fieldNameInRight"`, or `on="fieldName, otherFieldName=rightOtherFieldName"`.
+
+=== outerHashJoin Syntax
+
+[source,text]
+----
+outerHashJoin(
+ search(people, q=*:*, fl="personId,name", sort="personId asc"),
+ hashed=search(pets, q=type:cat, fl="personId,petName", sort="personId asc"),
+ on="personId"
+)
+
+outerHashJoin(
+ search(people, q=*:*, fl="personId,name", sort="personId asc"),
+ hashed=search(pets, q=type:cat, fl="ownerId,petName", sort="ownerId asc"),
+ on="personId=ownerId"
+)
+
+outerHashJoin(
+ search(people, q=*:*, fl="personId,name", sort="personId asc"),
+ hashed=select(
+ search(pets, q=type:cat, fl="ownerId,name", sort="ownerId asc"),
+ ownerId,
+ name as petName
+ ),
+ on="personId=ownerId"
+)
+----
+
+== parallel
+
+The `parallel` function wraps a streaming expression and sends it to N worker nodes to be processed in parallel.
+
+The parallel function requires that the `partitionKeys` parameter be provided to the underlying searches. The `partitionKeys` parameter will partition the search results (tuples) across the worker nodes. Tuples with the same values in the partitionKeys field will be shuffled to the same worker nodes.
+
+The parallel function maintains the sort order of the tuples returned by the worker nodes, so the sort criteria of the parallel function must match up with the sort order of the tuples returned by the workers.
+
+.Worker Collections
+[TIP]
+====
+The worker nodes can be from the same collection as the data, or they can be a different collection entirely, even one that only exists for parallel streaming expressions. A worker collection can be any SolrCloud collection that has the `/stream` handler configured. Unlike normal SolrCloud collections, worker collections don't have to hold any data. Worker collections can be empty collections that exist only to execute streaming expressions.
+====
+
+=== parallel Parameters
+
+* `collection`: Name of the worker collection to send the StreamExpression to.
+* `StreamExpression`: Expression to send to the worker collection.
+* `workers`: Number of workers in the worker collection to send the expression to.
+* `zkHost`: (Optional) The ZooKeeper connect string where the worker collection resides.
+* `sort`: The sort criteria for ordering tuples returned by the worker nodes.
+
+=== parallel Syntax
+
+[source,text]
+----
+ parallel(workerCollection,
+ reduce(search(collection1, q=*:*, fl="id,a_s,a_i,a_f", sort="a_s desc", partitionKeys="a_s"),
+ by="a_s",
+ group(sort="a_f desc", n="4")),
+ workers="20",
+ zkHost="localhost:9983",
+ sort="a_s desc")
+----
+
+The expression above shows a `parallel` function wrapping a `reduce` function. This will cause the `reduce` function to be run in parallel across 20 worker nodes.
+
+== priority
+
+The `priority` function is a simple priority scheduler for the <<executor>> function. The `executor` function doesn't directly have a concept of task prioritization; instead it simply executes tasks in the order that they are read from it's underlying stream. The `priority` function provides the ability to schedule a higher priority task ahead of lower priority tasks that were submitted earlier.
+
+The `priority` function wraps two <<stream-source-reference.adoc#topic,topics>> that are both emitting tuples that contain streaming expressions to execute. The first topic is considered the higher priority task queue.
+
+Each time the `priority` function is called, it checks the higher priority task queue to see if there are any tasks to execute. If tasks are waiting in the higher priority queue then the priority function will emit the higher priority tasks. If there are no high priority tasks to run, the lower priority queue tasks are emitted.
+
+The `priority` function will only emit a batch of tasks from one of the queues each time it is called. This ensures that no lower priority tasks are executed until the higher priority queue has no tasks to run.
+
+=== priority Parameters
+
+* `topic expression`: (Mandatory) the high priority task queue
+* `topic expression`: (Mandatory) the lower priority task queue
+
+=== priority Syntax
+
+[source,text]
+----
+daemon(id="myDaemon",
+ executor(threads=10,
+ priority(topic(checkpointCollection, storedExpressions, q="priority:high", fl="id, expr_s", initialCheckPoint=0,id="highPriorityTasks"),
+ topic(checkpointCollection, storedExpressions, q="priority:low", fl="id, expr_s", initialCheckPoint=0,id="lowPriorityTasks"))))
+----
+
+In the example above the `daemon` function is calling the executor iteratively. Each time it's called, the `executor` function will execute the tasks emitted by the `priority` function. The `priority` function wraps two topics. The first topic is the higher priority task queue, the second topics is the lower priority topic.
+
+== reduce
+
+The `reduce` function wraps an internal stream and groups tuples by common fields.
+
+Each tuple group is operated on as a single block by a pluggable reduce operation. The group operation provided with Solr implements distributed grouping functionality. The group operation also serves as an example reduce operation that can be referred to when building custom reduce operations.
+
+[IMPORTANT]
+====
+The reduce function relies on the sort order of the underlying stream. Accordingly the sort order of the underlying stream must be aligned with the group by field.
+====
+
+=== reduce Parameters
+
+* `StreamExpression`: (Mandatory)
+* `by`: (Mandatory) A comma separated list of fields to group by.
+* `Reduce Operation`: (Mandatory)
+
+=== reduce Syntax
+
+[source,text]
+----
+reduce(search(collection1, q=*:*, fl="id,a_s,a_i,a_f", sort="a_s asc, a_f asc"),
+ by="a_s",
+ group(sort="a_f desc", n="4")
+)
+----
+
+== rollup
+
+The `rollup` function wraps another stream function and rolls up aggregates over bucket fields. The rollup function relies on the sort order of the underlying stream to rollup aggregates one grouping at a time. Accordingly, the sort order of the underlying stream must match the fields in the `over` parameter of the rollup function.
+
+The rollup function also needs to process entire result sets in order to perform its aggregations. When the underlying stream is the `search` function, the `/export` handler can be used to provide full sorted result sets to the rollup function. This sorted approach allows the rollup function to perform aggregations over very high cardinality fields. The disadvantage of this approach is that the tuples must be sorted and streamed across the network to a worker node to be aggregated. For faster aggregation over low to moderate cardinality fields, the `facet` function can be used.
+
+=== rollup Parameters
+
+* `StreamExpression` (Mandatory)
+* `over`: (Mandatory) A list of fields to group by.
+* `metrics`: (Mandatory) The list of metrics to compute. Currently supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)`, `count(*)`.
+
+=== rollup Syntax
+
+[source,text]
+----
+rollup(
+ search(collection1, q=*:*, fl="a_s,a_i,a_f", qt="/export", sort="a_s asc"),
+ over="a_s",
+ sum(a_i),
+ sum(a_f),
+ min(a_i),
+ min(a_f),
+ max(a_i),
+ max(a_f),
+ avg(a_i),
+ avg(a_f),
+ count(*)
+)
+----
+
+The example about shows the rollup function wrapping the search function. Notice that search function is using the `/export` handler to provide the entire result set to the rollup stream. Also notice that the search function's *sort param* matches up with the rollup's `over` parameter. This allows the rollup function to rollup the over the `a_s` field, one group at a time.
+
+== scoreNodes
+
+See section in <<graph-traversal.adoc#using-the-scorenodes-function-to-make-a-recommendation,graph traversal>>.
+
+== select
+
+The `select` function wraps a streaming expression and outputs tuples containing a subset or modified set of fields from the incoming tuples. The list of fields included in the output tuple can contain aliases to effectively rename fields. The `select` stream supports both operations and evaluators. One can provide a list of operations and evaluators to perform on any fields, such as `replace, add, if`, etc....
+
+=== select Parameters
+
+* `StreamExpression`
+* `fieldName`: name of field to include in the output tuple (can include multiple of these), such as `outputTuple[fieldName] = inputTuple[fieldName]`
+* `fieldName as aliasFieldName`: aliased field name to include in the output tuple (can include multiple of these), such as `outputTuple[aliasFieldName] = incomingTuple[fieldName]`
+* `replace(fieldName, value, withValue=replacementValue)`: if `incomingTuple[fieldName] == value` then `outgoingTuple[fieldName]` will be set to `replacementValue`. `value` can be the string "null" to replace a null value with some other value.
+* `replace(fieldName, value, withField=otherFieldName)`: if `incomingTuple[fieldName] == value` then `outgoingTuple[fieldName]` will be set to the value of `incomingTuple[otherFieldName]`. `value` can be the string "null" to replace a null value with some other value.
+
+=== select Syntax
+
+[source,text]
+----
+// output tuples with fields teamName, wins, losses, and winPercentages where a null value for wins or losses is translated to the value of 0
+select(
+ search(collection1, fl="id,teamName_s,wins,losses", q="*:*", sort="id asc"),
+ teamName_s as teamName,
+ wins,
+ losses,
+ replace(wins,null,withValue=0),
+ replace(losses,null,withValue=0),
+ if(eq(0,wins), 0, div(add(wins,losses), wins)) as winPercentage
+)
+----
+
+== sort
+
+The `sort` function wraps a streaming expression and re-orders the tuples. The sort function emits all incoming tuples in the new sort order. The sort function reads all tuples from the incoming stream, re-orders them using an algorithm with `O(nlog(n))` performance characteristics, where n is the total number of tuples in the incoming stream, and then outputs the tuples in the new sort order. Because all tuples are read into memory, the memory consumption of this function grows linearly with the number of tuples in the incoming stream.
+
+=== sort Parameters
+
+* `StreamExpression`
+* `by`: Sort criteria for re-ordering the tuples
+
+=== sort Syntax
+
+The expression below finds dog owners and orders the results by owner and pet name. Notice that it uses an efficient innerJoin by first ordering by the person/owner id and then re-orders the final output by the owner and pet names.
+
+[source,text]
+----
+sort(
+ innerJoin(
+ search(people, q=*:*, fl="id,name", sort="id asc"),
+ search(pets, q=type:dog, fl="owner,petName", sort="owner asc"),
+ on="id=owner"
+ ),
+ by="name asc, petName asc"
+)
+----
+
+== top
+
+The `top` function wraps a streaming expression and re-orders the tuples. The top function emits only the top N tuples in the new sort order. The top function re-orders the underlying stream so the sort criteria *does not* have to match up with the underlying stream.
+
+=== top Parameters
+
+* `n`: Number of top tuples to return.
+* `StreamExpression`
+* `sort`: Sort criteria for selecting the top N tuples.
+
+=== top Syntax
+
+The expression below finds the top 3 results of the underlying search. Notice that it reverses the sort order. The top function re-orders the results of the underlying stream.
+
+[source,text]
+----
+top(n=3,
+ search(collection1,
+ q="*:*",
+ qt="/export",
+ fl="id,a_s,a_i,a_f",
+ sort="a_f desc, a_i desc"),
+ sort="a_f asc, a_i asc")
+----
+
+== unique
+
+The `unique` function wraps a streaming expression and emits a unique stream of tuples based on the `over` parameter. The unique function relies on the sort order of the underlying stream. The `over` parameter must match up with the sort order of the underlying stream.
+
+The unique function implements a non-co-located unique algorithm. This means that records with the same unique `over` field do not need to be co-located on the same shard. When executed in the parallel, the `partitionKeys` parameter must be the same as the unique `over` field so that records with the same keys will be shuffled to the same worker.
+
+=== unique Parameters
+
+* `StreamExpression`
+* `over`: The unique criteria.
+
+=== unique Syntax
+
+[source,text]
+----
+unique(
+ search(collection1,
+ q="*:*",
+ qt="/export",
+ fl="id,a_s,a_i,a_f",
+ sort="a_f asc, a_i asc"),
+ over="a_f")
+----
+
+== update
+
+The `update` function wraps another functions and sends the tuples to a SolrCloud collection for indexing.
+
+=== update Parameters
+
+* `destinationCollection`: (Mandatory) The collection where the tuples will indexed.
+* `batchSize`: (Mandatory) The indexing batch size.
+* `StreamExpression`: (Mandatory)
+
+=== update Syntax
+
+[source,text]
+----
+ update(destinationCollection,
+ batchSize=500,
+ search(collection1,
+ q=*:*,
+ fl="id,a_s,a_i,a_f,s_multi,i_multi",
+ sort="a_f asc, a_i asc"))
+
+----
+
+The example above sends the tuples returned by the `search` function to the `destinationCollection` to be indexed.
[33/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11406: Solr 7.0
cannot read indexes from 6.x versions
Posted by no...@apache.org.
SOLR-11406: Solr 7.0 cannot read indexes from 6.x versions
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/39ba1c3d
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/39ba1c3d
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/39ba1c3d
Branch: refs/heads/feature/autoscaling
Commit: 39ba1c3da56c5ac3e9476830efe1f4c396ccf7c0
Parents: c322e36
Author: Steve Rowe <sa...@gmail.com>
Authored: Fri Sep 29 16:26:59 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Fri Sep 29 16:33:07 2017 -0400
----------------------------------------------------------------------
.../lucene/backward-codecs/pom.xml.template | 13 +++
.../maven/solr/core/src/test/pom.xml.template | 7 ++
.../index/TestBackwardsCompatibility.java | 12 ++-
solr/CHANGES.txt | 21 +++++
solr/common-build.xml | 6 ++
solr/core/build.xml | 3 +-
.../solr/index/SlowCompositeReaderWrapper.java | 16 ++--
.../solr/configsets/backcompat/conf/schema.xml | 22 +++++
.../configsets/backcompat/conf/solrconfig.xml | 43 +++++++++
.../backcompat/TestLuceneIndexBackCompat.java | 96 ++++++++++++++++++++
10 files changed, 230 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/dev-tools/maven/lucene/backward-codecs/pom.xml.template
----------------------------------------------------------------------
diff --git a/dev-tools/maven/lucene/backward-codecs/pom.xml.template b/dev-tools/maven/lucene/backward-codecs/pom.xml.template
index e40148f..f6e6680 100644
--- a/dev-tools/maven/lucene/backward-codecs/pom.xml.template
+++ b/dev-tools/maven/lucene/backward-codecs/pom.xml.template
@@ -71,5 +71,18 @@
</excludes>
</testResource>
</testResources>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-jar-plugin</artifactId>
+ <executions>
+ <execution>
+ <goals>
+ <goal>test-jar</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
</build>
</project>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/dev-tools/maven/solr/core/src/test/pom.xml.template
----------------------------------------------------------------------
diff --git a/dev-tools/maven/solr/core/src/test/pom.xml.template b/dev-tools/maven/solr/core/src/test/pom.xml.template
index a4e979f..fd6b8e7 100644
--- a/dev-tools/maven/solr/core/src/test/pom.xml.template
+++ b/dev-tools/maven/solr/core/src/test/pom.xml.template
@@ -61,6 +61,13 @@
<type>test-jar</type>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.lucene</groupId>
+ <artifactId>lucene-backward-codecs</artifactId>
+ <version>${project.version}</version>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
@solr-core.internal.test.dependencies@
@solr-core.external.test.dependencies@
</dependencies>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
index abb189a..c202973 100644
--- a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
+++ b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
@@ -277,7 +277,11 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
final static String[] oldNames = {
};
-
+
+ public static String[] getOldNames() {
+ return oldNames;
+ }
+
final String[] unsupportedNames = {
"1.9.0-cfs",
"1.9.0-nocfs",
@@ -440,7 +444,11 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
// TODO: on 6.0.0 release, gen the single segment indices and add here:
final static String[] oldSingleSegmentNames = {
};
-
+
+ public static String[] getOldSingleSegmentNames() {
+ return oldSingleSegmentNames;
+ }
+
static Map<String,Directory> oldIndexDirs;
/**
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 13cee84..aed0c22 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -235,6 +235,27 @@ Other Changes
* SOLR-10451: Remove contrib/ltr/lib from lib includes in the techproducts example config (sungjunyoung via janhoy)
+================== 7.0.1 ==================
+
+Consult the LUCENE_CHANGES.txt file for additional, low level, changes in this release.
+
+Versions of Major Components
+---------------------
+Apache Tika 1.13
+Carrot2 3.15.0
+Velocity 1.7 and Velocity Tools 2.0
+Apache UIMA 2.3.1
+Apache ZooKeeper 3.4.10
+Jetty 9.3.14.v20161028
+
+Bug Fixes
+----------------------
+
+* SOLR-11297: Message "Lock held by this virtual machine" during startup. Solr is trying to start some cores twice.
+ (Luiz Armesto, Shawn Heisey, Erick Erickson)
+
+* SOLR-11406: Solr 7.0 cannot read indexes from 6.x versions. (Shawn Heisey, Steve Rowe)
+
================== 7.0.0 ==================
Versions of Major Components
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/solr/common-build.xml
----------------------------------------------------------------------
diff --git a/solr/common-build.xml b/solr/common-build.xml
index 88f71ef..ec37848 100644
--- a/solr/common-build.xml
+++ b/solr/common-build.xml
@@ -493,6 +493,12 @@
</ant>
</target>
+ <target name="-compile-test-lucene-backward-codecs">
+ <ant dir="${common.dir}/backward-codecs" target="compile-test" inheritAll="false">
+ <propertyset refid="uptodate.and.compiled.properties"/>
+ </ant>
+ </target>
+
<!-- Solr contrib targets -->
<target name="-compile-analysis-extras">
<ant dir="${common-solr.dir}/contrib/analysis-extras" target="compile" inheritAll="false">
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/solr/core/build.xml
----------------------------------------------------------------------
diff --git a/solr/core/build.xml b/solr/core/build.xml
index f7a2a59..45bd1b4 100644
--- a/solr/core/build.xml
+++ b/solr/core/build.xml
@@ -31,7 +31,7 @@
<target name="compile-core" depends="compile-solrj,common-solr.compile-core"/>
- <target name="compile-test" depends="jar-analyzers-icu,-compile-test-lucene-queryparser,-compile-analysis-extras,common-solr.compile-test"/>
+ <target name="compile-test" depends="jar-analyzers-icu,-compile-test-lucene-queryparser,-compile-test-lucene-backward-codecs,-compile-analysis-extras,common-solr.compile-test"/>
<path id="test.classpath">
<path refid="solr.test.base.classpath"/>
@@ -39,6 +39,7 @@
<pathelement location="${analyzers-icu.jar}"/>
<pathelement location="${common-solr.dir}/build/contrib/solr-analysis-extras/classes/java"/>
<pathelement location="${common.dir}/build/queryparser/classes/test"/>
+ <pathelement location="${common.dir}/build/backward-codecs/classes/test"/>
<fileset dir="${common-solr.dir}/contrib/analysis-extras/lib" includes="icu4j*.jar"/>
</path>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java b/solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java
index 2d612e9..283d245 100644
--- a/solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java
+++ b/solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java
@@ -69,12 +69,16 @@ public final class SlowCompositeReaderWrapper extends LeafReader {
if (reader.leaves().isEmpty()) {
metaData = new LeafMetaData(Version.LATEST.major, Version.LATEST, null);
} else {
- Version minVersion = reader.leaves().stream()
- .map(LeafReaderContext::reader)
- .map(LeafReader::getMetaData)
- .map(LeafMetaData::getMinVersion)
- .reduce((v1, v2) -> v1 == null ? null : v2 == null ? null : v2.onOrAfter(v1) ? v1 : v2)
- .get();
+ Version minVersion = Version.LATEST;
+ for (LeafReaderContext leafReaderContext : reader.leaves()) {
+ Version leafVersion = leafReaderContext.reader().getMetaData().getMinVersion();
+ if (leafVersion == null) {
+ minVersion = null;
+ break;
+ } else if (minVersion.onOrAfter(leafVersion)) {
+ minVersion = leafVersion;
+ }
+ }
metaData = new LeafMetaData(reader.leaves().get(0).reader().getMetaData().getCreatedVersionMajor(), minVersion, null);
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/solr/core/src/test-files/solr/configsets/backcompat/conf/schema.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/backcompat/conf/schema.xml b/solr/core/src/test-files/solr/configsets/backcompat/conf/schema.xml
new file mode 100644
index 0000000..b49edd2
--- /dev/null
+++ b/solr/core/src/test-files/solr/configsets/backcompat/conf/schema.xml
@@ -0,0 +1,22 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<schema name="minimal" version="1.1">
+ <fieldType name="string" class="solr.StrField"/>
+ <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
+ <uniquekey>id</uniquekey>
+</schema>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/solr/core/src/test-files/solr/configsets/backcompat/conf/solrconfig.xml
----------------------------------------------------------------------
diff --git a/solr/core/src/test-files/solr/configsets/backcompat/conf/solrconfig.xml b/solr/core/src/test-files/solr/configsets/backcompat/conf/solrconfig.xml
new file mode 100644
index 0000000..f82d5f4
--- /dev/null
+++ b/solr/core/src/test-files/solr/configsets/backcompat/conf/solrconfig.xml
@@ -0,0 +1,43 @@
+<?xml version="1.0" ?>
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!-- Minimal solrconfig.xml with /select, /admin and /update only -->
+
+<config>
+ <dataDir>${solr.data.dir:}</dataDir>
+
+ <directoryFactory name="DirectoryFactory" class="solr.StandardDirectoryFactory"/>
+
+ <schemaFactory class="ClassicIndexSchemaFactory"/>
+
+ <luceneMatchVersion>${tests.luceneMatchVersion:LATEST}</luceneMatchVersion>
+
+ <updateHandler class="solr.DirectUpdateHandler2">
+ <commitWithin>
+ <softCommit>${solr.commitwithin.softcommit:true}</softCommit>
+ </commitWithin>
+ </updateHandler>
+
+ <requestHandler name="/select" class="solr.SearchHandler">
+ <lst name="defaults">
+ <str name="echoParams">explicit</str>
+ <str name="indent">true</str>
+ </lst>
+ </requestHandler>
+</config>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/39ba1c3d/solr/core/src/test/org/apache/solr/backcompat/TestLuceneIndexBackCompat.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/backcompat/TestLuceneIndexBackCompat.java b/solr/core/src/test/org/apache/solr/backcompat/TestLuceneIndexBackCompat.java
new file mode 100644
index 0000000..2541d1f
--- /dev/null
+++ b/solr/core/src/test/org/apache/solr/backcompat/TestLuceneIndexBackCompat.java
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.backcompat;
+
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.lucene.index.TestBackwardsCompatibility;
+import org.apache.lucene.util.TestUtil;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.util.TestHarness;
+import org.junit.Test;
+
+/** Verify we can read/write previous versions' Lucene indexes. */
+public class TestLuceneIndexBackCompat extends SolrTestCaseJ4 {
+ private static final String[] oldNames = TestBackwardsCompatibility.getOldNames();
+ private static final String[] oldSingleSegmentNames = TestBackwardsCompatibility.getOldSingleSegmentNames();
+
+ @Test
+ public void testOldIndexes() throws Exception {
+ List<String> names = new ArrayList<>(oldNames.length + oldSingleSegmentNames.length);
+ names.addAll(Arrays.asList(oldNames));
+ names.addAll(Arrays.asList(oldSingleSegmentNames));
+ for (String name : names) {
+ setupCore(name);
+
+ assertQ(req("q", "*:*", "rows", "0"), "//result[@numFound='35']");
+
+ assertU(adoc("id", "id_123456789"));
+ assertU(commit());
+
+ deleteCore();
+ }
+ }
+
+ private void setupCore(String coreName) throws Exception {
+ if (h != null) {
+ h.close();
+ }
+ Path solrHome = createTempDir(coreName).toAbsolutePath();
+ Files.createDirectories(solrHome);
+ Path coreDir = solrHome.resolve(coreName);
+ Path confDir = coreDir.resolve("conf");
+ Files.createDirectories(confDir);
+ Path dataDir = coreDir.resolve("data");
+ Path indexDir = dataDir.resolve("index");
+ Files.createDirectories(indexDir);
+
+ Files.copy(getFile("solr/solr.xml").toPath(), solrHome.resolve("solr.xml"));
+ FileUtils.copyDirectory(configset("backcompat").toFile(), confDir.toFile());
+
+ try (Writer writer = new OutputStreamWriter(Files.newOutputStream(coreDir.resolve("core.properties")), StandardCharsets.UTF_8)) {
+ Properties coreProps = new Properties();
+ coreProps.put("name", coreName);
+ coreProps.store(writer, null);
+ }
+
+ InputStream resource = TestBackwardsCompatibility.class.getResourceAsStream("index." + coreName + ".zip");
+ assertNotNull("Index name " + coreName + " not found", resource);
+ TestUtil.unzip(resource, indexDir);
+
+ configString = "solrconfig.xml";
+ schemaString = "schema.xml";
+ testSolrHome = solrHome;
+ System.setProperty("solr.solr.home", solrHome.toString());
+ ignoreException("ignore_exception");
+ solrConfig = TestHarness.createConfig(testSolrHome, coreName, getSolrConfigFile());
+ h = new TestHarness(coreName, dataDir.toString(), solrConfig, getSchemaFile());
+ lrf = h.getRequestFactory("",0,20, CommonParams.VERSION,"2.2");
+ }
+}
[40/50] [abbrv] lucene-solr:feature/autoscaling: LUCENE-7974: Add
N-dimensional FloatPoint K-nearest-neighbor implementation
Posted by no...@apache.org.
LUCENE-7974: Add N-dimensional FloatPoint K-nearest-neighbor implementation
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/d52564c0
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/d52564c0
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/d52564c0
Branch: refs/heads/feature/autoscaling
Commit: d52564c079bb7ca842a1041bc8baff468e1938d4
Parents: 472d520
Author: Steve Rowe <sa...@apache.org>
Authored: Sun Oct 1 18:50:06 2017 -0400
Committer: Steve Rowe <sa...@apache.org>
Committed: Sun Oct 1 18:50:55 2017 -0400
----------------------------------------------------------------------
lucene/CHANGES.txt | 3 +
.../document/FloatPointNearestNeighbor.java | 382 +++++++++++++++++++
.../document/TestFloatPointNearestNeighbor.java | 239 ++++++++++++
3 files changed, 624 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/d52564c0/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 16482e9..e0c8124 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -34,6 +34,9 @@ New Features
* LUCENE-7973: Update dictionary version for Ukrainian analyzer to 3.9.0 (Andriy
Rysin via Dawid Weiss)
+
+* LUCENE-7974: Add FloatPointNearestNeighbor, an N-dimensional FloatPoint
+ K-nearest-neighbor search implementation. (Steve Rowe)
Optimizations
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/d52564c0/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
----------------------------------------------------------------------
diff --git a/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java b/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
new file mode 100644
index 0000000..d3360a8
--- /dev/null
+++ b/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.document;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.PriorityQueue;
+
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.PointValues;
+import org.apache.lucene.search.FieldDoc;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.search.TopFieldDocs;
+import org.apache.lucene.util.Bits;
+import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.bkd.BKDReader;
+
+/**
+ * KNN search on top of N dimensional indexed float points.
+ *
+ * @lucene.experimental
+ */
+public class FloatPointNearestNeighbor {
+
+ static class Cell implements Comparable<Cell> {
+ final int readerIndex;
+ final byte[] minPacked;
+ final byte[] maxPacked;
+ final BKDReader.IndexTree index;
+
+ /** The closest possible distance^2 of all points in this cell */
+ final double distanceSquared;
+
+ Cell(BKDReader.IndexTree index, int readerIndex, byte[] minPacked, byte[] maxPacked, double distanceSquared) {
+ this.index = index;
+ this.readerIndex = readerIndex;
+ this.minPacked = minPacked.clone();
+ this.maxPacked = maxPacked.clone();
+ this.distanceSquared = distanceSquared;
+ }
+
+ public int compareTo(Cell other) {
+ return Double.compare(distanceSquared, other.distanceSquared);
+ }
+
+ @Override
+ public String toString() {
+ return "Cell(readerIndex=" + readerIndex + " nodeID=" + index.getNodeID()
+ + " isLeaf=" + index.isLeafNode() + " distanceSquared=" + distanceSquared + ")";
+ }
+ }
+
+ private static class NearestVisitor implements PointValues.IntersectVisitor {
+ int curDocBase;
+ Bits curLiveDocs;
+ final int topN;
+ final PriorityQueue<NearestHit> hitQueue;
+ final float[] origin;
+ private int dims;
+ private int updateMinMaxCounter;
+ private float[] min;
+ private float[] max;
+
+
+ public NearestVisitor(PriorityQueue<NearestHit> hitQueue, int topN, float[] origin) {
+ this.hitQueue = hitQueue;
+ this.topN = topN;
+ this.origin = origin;
+ dims = origin.length;
+ min = new float[dims];
+ max = new float[dims];
+ Arrays.fill(min, Float.NEGATIVE_INFINITY);
+ Arrays.fill(max, Float.POSITIVE_INFINITY);
+ }
+
+ @Override
+ public void visit(int docID) {
+ throw new AssertionError();
+ }
+
+ private static final int MANTISSA_BITS = 23;
+
+ /**
+ * Returns the minimum value that will change the given distance when added to it.
+ *
+ * This value is calculated from the distance exponent reduced by (at most) 23,
+ * the number of bits in a float mantissa. This is necessary when the result of
+ * subtracting/adding the distance in a single dimension has an exponent that
+ * differs significantly from that of the distance value. Without this fudge
+ * factor (i.e. only subtracting/adding the distance), cells and values can be
+ * inappropriately judged as outside the search radius.
+ */
+ private float getMinDelta(float distance) {
+ int exponent = Float.floatToIntBits(distance) >> MANTISSA_BITS; // extract biased exponent (distance is positive)
+ if (exponent == 0) {
+ return Float.MIN_VALUE;
+ } else {
+ exponent = exponent <= MANTISSA_BITS ? 1 : exponent - MANTISSA_BITS; // Avoid underflow
+ return Float.intBitsToFloat(exponent << MANTISSA_BITS);
+ }
+ }
+
+ private void maybeUpdateMinMax() {
+ if (updateMinMaxCounter < 1024 || (updateMinMaxCounter & 0x3F) == 0x3F) {
+ NearestHit hit = hitQueue.peek();
+ float distance = (float)Math.sqrt(hit.distanceSquared);
+ float minDelta = getMinDelta(distance);
+ // String oldMin = Arrays.toString(min);
+ // String oldMax = Arrays.toString(max);
+ for (int d = 0 ; d < dims ; ++d) {
+ min[d] = (origin[d] - distance) - minDelta;
+ max[d] = (origin[d] + distance) + minDelta;
+ // System.out.println("origin[" + d + "] (" + origin[d] + ") - distance (" + distance + ") - minDelta (" + minDelta + ") = min[" + d + "] (" + min[d] + ")");
+ // System.out.println("origin[" + d + "] (" + origin[d] + ") + distance (" + distance + ") + minDelta (" + minDelta + ") = max[" + d + "] (" + max[d] + ")");
+ }
+ // System.out.println("maybeUpdateMinMax: min: " + oldMin + " -> " + Arrays.toString(min) + " max: " + oldMax + " -> " + Arrays.toString(max));
+ }
+ ++updateMinMaxCounter;
+ }
+
+ @Override
+ public void visit(int docID, byte[] packedValue) {
+ // System.out.println("visit docID=" + docID + " liveDocs=" + curLiveDocs);
+
+ if (curLiveDocs != null && curLiveDocs.get(docID) == false) {
+ return;
+ }
+
+ float[] docPoint = new float[dims];
+ for (int d = 0, offset = 0 ; d < dims ; ++d, offset += Float.BYTES) {
+ docPoint[d] = FloatPoint.decodeDimension(packedValue, offset);
+ if (docPoint[d] > max[d] || docPoint[d] < min[d]) {
+
+ // if (docPoint[d] > max[d]) {
+ // System.out.println(" skipped because docPoint[" + d + "] (" + docPoint[d] + ") > max[" + d + "] (" + max[d] + ")");
+ // } else {
+ // System.out.println(" skipped because docPoint[" + d + "] (" + docPoint[d] + ") < min[" + d + "] (" + min[d] + ")");
+ // }
+
+ return;
+ }
+ }
+
+ double distanceSquared = euclideanDistanceSquared(origin, docPoint);
+
+ // System.out.println(" visit docID=" + docID + " distanceSquared=" + distanceSquared + " value: " + Arrays.toString(docPoint));
+
+ int fullDocID = curDocBase + docID;
+
+ if (hitQueue.size() == topN) { // queue already full
+ NearestHit bottom = hitQueue.peek();
+ // System.out.println(" bottom distanceSquared=" + bottom.distanceSquared);
+ if (distanceSquared < bottom.distanceSquared
+ // we don't collect docs in order here, so we must also test the tie-break case ourselves:
+ || (distanceSquared == bottom.distanceSquared && fullDocID < bottom.docID)) {
+ hitQueue.poll();
+ bottom.docID = fullDocID;
+ bottom.distanceSquared = distanceSquared;
+ hitQueue.offer(bottom);
+ // System.out.println(" ** keep1, now bottom=" + bottom);
+ maybeUpdateMinMax();
+ }
+ } else {
+ NearestHit hit = new NearestHit();
+ hit.docID = fullDocID;
+ hit.distanceSquared = distanceSquared;
+ hitQueue.offer(hit);
+ // System.out.println(" ** keep2, new addition=" + hit);
+ }
+ }
+
+ @Override
+ public PointValues.Relation compare(byte[] minPackedValue, byte[] maxPackedValue) {
+ throw new AssertionError();
+ }
+ }
+
+ static class NearestHit {
+ public int docID;
+ public double distanceSquared;
+
+ @Override
+ public String toString() {
+ return "NearestHit(docID=" + docID + " distanceSquared=" + distanceSquared + ")";
+ }
+ }
+
+ public static NearestHit[] nearest(List<BKDReader> readers, List<Bits> liveDocs, List<Integer> docBases, final int topN, float[] origin) throws IOException {
+
+ // System.out.println("NEAREST: readers=" + readers + " liveDocs=" + liveDocs + " origin: " + Arrays.toString(origin));
+
+ // Holds closest collected points seen so far:
+ // TODO: if we used lucene's PQ we could just updateTop instead of poll/offer:
+ final PriorityQueue<NearestHit> hitQueue = new PriorityQueue<>(topN, (a, b) -> {
+ // sort by opposite distance natural order
+ int cmp = Double.compare(a.distanceSquared, b.distanceSquared);
+ return cmp != 0 ? -cmp : b.docID - a.docID; // tie-break by higher docID
+ });
+
+ // Holds all cells, sorted by closest to the point:
+ PriorityQueue<Cell> cellQueue = new PriorityQueue<>();
+
+ NearestVisitor visitor = new NearestVisitor(hitQueue, topN, origin);
+ List<BKDReader.IntersectState> states = new ArrayList<>();
+
+ // Add root cell for each reader into the queue:
+ int bytesPerDim = -1;
+
+ for (int i = 0 ; i < readers.size() ; ++i) {
+ BKDReader reader = readers.get(i);
+ if (bytesPerDim == -1) {
+ bytesPerDim = reader.getBytesPerDimension();
+ } else if (bytesPerDim != reader.getBytesPerDimension()) {
+ throw new IllegalStateException("bytesPerDim changed from " + bytesPerDim
+ + " to " + reader.getBytesPerDimension() + " across readers");
+ }
+ byte[] minPackedValue = reader.getMinPackedValue();
+ byte[] maxPackedValue = reader.getMaxPackedValue();
+ BKDReader.IntersectState state = reader.getIntersectState(visitor);
+ states.add(state);
+
+ cellQueue.offer(new Cell(state.index, i, reader.getMinPackedValue(), reader.getMaxPackedValue(),
+ approxBestDistanceSquared(minPackedValue, maxPackedValue, origin)));
+ }
+
+ LOOP_OVER_CELLS: while (cellQueue.size() > 0) {
+ Cell cell = cellQueue.poll();
+ // System.out.println(" visit " + cell);
+
+ // TODO: if we replace approxBestDistance with actualBestDistance, we can put an opto here to break once this "best" cell is fully outside of the hitQueue bottom's radius:
+ BKDReader reader = readers.get(cell.readerIndex);
+
+ if (cell.index.isLeafNode()) {
+ // System.out.println(" leaf");
+ // Leaf block: visit all points and possibly collect them:
+ visitor.curDocBase = docBases.get(cell.readerIndex);
+ visitor.curLiveDocs = liveDocs.get(cell.readerIndex);
+ reader.visitLeafBlockValues(cell.index, states.get(cell.readerIndex));
+ // System.out.println(" now " + hitQueue.size() + " hits");
+ } else {
+ // System.out.println(" non-leaf");
+ // Non-leaf block: split into two cells and put them back into the queue:
+
+ if (hitQueue.size() == topN) {
+ for (int d = 0, offset = 0; d < visitor.dims; ++d, offset += Float.BYTES) {
+ float cellMaxAtDim = FloatPoint.decodeDimension(cell.maxPacked, offset);
+ float cellMinAtDim = FloatPoint.decodeDimension(cell.minPacked, offset);
+ if (cellMaxAtDim < visitor.min[d] || cellMinAtDim > visitor.max[d]) {
+ // this cell is outside our search radius; don't bother exploring any more
+
+ // if (cellMaxAtDim < visitor.min[d]) {
+ // System.out.println(" skipped because cell max at " + d + " (" + cellMaxAtDim + ") < visitor.min[" + d + "] (" + visitor.min[d] + ")");
+ // } else {
+ // System.out.println(" skipped because cell min at " + d + " (" + cellMinAtDim + ") > visitor.max[" + d + "] (" + visitor.max[d] + ")");
+ // }
+
+ continue LOOP_OVER_CELLS;
+ }
+ }
+ }
+ BytesRef splitValue = BytesRef.deepCopyOf(cell.index.getSplitDimValue());
+ int splitDim = cell.index.getSplitDim();
+
+ // we must clone the index so that we we can recurse left and right "concurrently":
+ BKDReader.IndexTree newIndex = cell.index.clone();
+ byte[] splitPackedValue = cell.maxPacked.clone();
+ System.arraycopy(splitValue.bytes, splitValue.offset, splitPackedValue, splitDim * bytesPerDim, bytesPerDim);
+
+ cell.index.pushLeft();
+ cellQueue.offer(new Cell(cell.index, cell.readerIndex, cell.minPacked, splitPackedValue,
+ approxBestDistanceSquared(cell.minPacked, splitPackedValue, origin)));
+
+ splitPackedValue = cell.minPacked.clone();
+ System.arraycopy(splitValue.bytes, splitValue.offset, splitPackedValue, splitDim * bytesPerDim, bytesPerDim);
+
+ newIndex.pushRight();
+ cellQueue.offer(new Cell(newIndex, cell.readerIndex, splitPackedValue, cell.maxPacked,
+ approxBestDistanceSquared(splitPackedValue, cell.maxPacked, origin)));
+ }
+ }
+
+ NearestHit[] hits = new NearestHit[hitQueue.size()];
+ int downTo = hitQueue.size()-1;
+ while (hitQueue.size() != 0) {
+ hits[downTo] = hitQueue.poll();
+ downTo--;
+ }
+ return hits;
+ }
+
+ private static double approxBestDistanceSquared(byte[] minPackedValue, byte[] maxPackedValue, float[] value) {
+ boolean insideCell = true;
+ float[] min = new float[value.length];
+ float[] max = new float[value.length];
+ double[] closest = new double[value.length];
+ for (int i = 0, offset = 0 ; i < value.length ; ++i, offset += Float.BYTES) {
+ min[i] = FloatPoint.decodeDimension(minPackedValue, offset);
+ max[i] = FloatPoint.decodeDimension(maxPackedValue, offset);
+ if (insideCell) {
+ if (value[i] < min[i] || value[i] > max[i]) {
+ insideCell = false;
+ }
+ }
+ double minDiff = Math.abs((double)value[i] - (double)min[i]);
+ double maxDiff = Math.abs((double)value[i] - (double)max[i]);
+ closest[i] = minDiff < maxDiff ? minDiff : maxDiff;
+ }
+ if (insideCell) {
+ return 0.0f;
+ }
+ double sumOfSquaredDiffs = 0.0d;
+ for (int d = 0 ; d < value.length ; ++d) {
+ sumOfSquaredDiffs += closest[d] * closest[d];
+ }
+ return sumOfSquaredDiffs;
+ }
+
+ static double euclideanDistanceSquared(float[] a, float[] b) {
+ double sumOfSquaredDifferences = 0.0d;
+ for (int d = 0 ; d < a.length ; ++d) {
+ double diff = (double)a[d] - (double)b[d];
+ sumOfSquaredDifferences += diff * diff;
+ }
+ return sumOfSquaredDifferences;
+ }
+
+ public static TopFieldDocs nearest(IndexSearcher searcher, String field, int topN, float... origin) throws IOException {
+ if (topN < 1) {
+ throw new IllegalArgumentException("topN must be at least 1; got " + topN);
+ }
+ if (field == null) {
+ throw new IllegalArgumentException("field must not be null");
+ }
+ if (searcher == null) {
+ throw new IllegalArgumentException("searcher must not be null");
+ }
+ List<BKDReader> readers = new ArrayList<>();
+ List<Integer> docBases = new ArrayList<>();
+ List<Bits> liveDocs = new ArrayList<>();
+ int totalHits = 0;
+ for (LeafReaderContext leaf : searcher.getIndexReader().leaves()) {
+ PointValues points = leaf.reader().getPointValues(field);
+ if (points != null) {
+ if (points instanceof BKDReader == false) {
+ throw new IllegalArgumentException("can only run on Lucene60PointsReader points implementation, but got " + points);
+ }
+ totalHits += points.getDocCount();
+ readers.add((BKDReader)points);
+ docBases.add(leaf.docBase);
+ liveDocs.add(leaf.reader().getLiveDocs());
+ }
+ }
+
+ NearestHit[] hits = nearest(readers, liveDocs, docBases, topN, origin);
+
+ // Convert to TopFieldDocs:
+ ScoreDoc[] scoreDocs = new ScoreDoc[hits.length];
+ for(int i=0;i<hits.length;i++) {
+ NearestHit hit = hits[i];
+ scoreDocs[i] = new FieldDoc(hit.docID, 0.0f, new Object[] { (float)Math.sqrt(hit.distanceSquared) });
+ }
+ return new TopFieldDocs(totalHits, scoreDocs, null, 0.0f);
+ }
+}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/d52564c0/lucene/sandbox/src/test/org/apache/lucene/document/TestFloatPointNearestNeighbor.java
----------------------------------------------------------------------
diff --git a/lucene/sandbox/src/test/org/apache/lucene/document/TestFloatPointNearestNeighbor.java b/lucene/sandbox/src/test/org/apache/lucene/document/TestFloatPointNearestNeighbor.java
new file mode 100644
index 0000000..9e3c3c1
--- /dev/null
+++ b/lucene/sandbox/src/test/org/apache/lucene/document/TestFloatPointNearestNeighbor.java
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.document;
+
+import java.util.Arrays;
+
+import org.apache.lucene.codecs.Codec;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.PointValues;
+import org.apache.lucene.index.RandomIndexWriter;
+import org.apache.lucene.index.SerialMergeScheduler;
+import org.apache.lucene.index.Term;
+import org.apache.lucene.search.FieldDoc;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.util.TestUtil;
+
+public class TestFloatPointNearestNeighbor extends LuceneTestCase {
+
+ public void testNearestNeighborWithDeletedDocs() throws Exception {
+ Directory dir = newDirectory();
+ RandomIndexWriter w = new RandomIndexWriter(random(), dir, getIndexWriterConfig());
+ Document doc = new Document();
+ doc.add(new FloatPoint("point", 40.0f, 50.0f));
+ doc.add(new StringField("id", "0", Field.Store.YES));
+ w.addDocument(doc);
+
+ doc = new Document();
+ doc.add(new FloatPoint("point", 45.0f, 55.0f));
+ doc.add(new StringField("id", "1", Field.Store.YES));
+ w.addDocument(doc);
+
+ DirectoryReader r = w.getReader();
+ // can't wrap because we require Lucene60PointsFormat directly but e.g. ParallelReader wraps with its own points impl:
+ IndexSearcher s = newSearcher(r, false);
+ FieldDoc hit = (FieldDoc)FloatPointNearestNeighbor.nearest(s, "point", 1, 40.0f, 50.0f).scoreDocs[0];
+ assertEquals("0", r.document(hit.doc).getField("id").stringValue());
+ r.close();
+
+ w.deleteDocuments(new Term("id", "0"));
+ r = w.getReader();
+ // can't wrap because we require Lucene60PointsFormat directly but e.g. ParallelReader wraps with its own points impl:
+ s = newSearcher(r, false);
+ hit = (FieldDoc)LatLonPoint.nearest(s, "point", 40.0, 50.0, 1).scoreDocs[0];
+ assertEquals("1", r.document(hit.doc).getField("id").stringValue());
+ r.close();
+ w.close();
+ dir.close();
+ }
+
+ public void testNearestNeighborWithAllDeletedDocs() throws Exception {
+ Directory dir = newDirectory();
+ RandomIndexWriter w = new RandomIndexWriter(random(), dir, getIndexWriterConfig());
+ Document doc = new Document();
+ doc.add(new FloatPoint("point", 40.0f, 50.0f));
+ doc.add(new StringField("id", "0", Field.Store.YES));
+ w.addDocument(doc);
+ doc = new Document();
+ doc.add(new FloatPoint("point", 45.0f, 55.0f));
+ doc.add(new StringField("id", "1", Field.Store.YES));
+ w.addDocument(doc);
+
+ DirectoryReader r = w.getReader();
+ // can't wrap because we require Lucene60PointsFormat directly but e.g. ParallelReader wraps with its own points impl:
+ IndexSearcher s = newSearcher(r, false);
+ FieldDoc hit = (FieldDoc)FloatPointNearestNeighbor.nearest(s, "point", 1, 40.0f, 50.0f).scoreDocs[0];
+ assertEquals("0", r.document(hit.doc).getField("id").stringValue());
+ r.close();
+
+ w.deleteDocuments(new Term("id", "0"));
+ w.deleteDocuments(new Term("id", "1"));
+ r = w.getReader();
+ // can't wrap because we require Lucene60PointsFormat directly but e.g. ParallelReader wraps with its own points impl:
+ s = newSearcher(r, false);
+ assertEquals(0, FloatPointNearestNeighbor.nearest(s, "point", 1, 40.0f, 50.0f).scoreDocs.length);
+ r.close();
+ w.close();
+ dir.close();
+ }
+
+ public void testTieBreakByDocID() throws Exception {
+ Directory dir = newDirectory();
+ IndexWriter w = new IndexWriter(dir, getIndexWriterConfig());
+ Document doc = new Document();
+ doc.add(new FloatPoint("point", 40.0f, 50.0f));
+ doc.add(new StringField("id", "0", Field.Store.YES));
+ w.addDocument(doc);
+ doc = new Document();
+ doc.add(new FloatPoint("point", 40.0f, 50.0f));
+ doc.add(new StringField("id", "1", Field.Store.YES));
+ w.addDocument(doc);
+
+ DirectoryReader r = DirectoryReader.open(w);
+ // can't wrap because we require Lucene60PointsFormat directly but e.g. ParallelReader wraps with its own points impl:
+ ScoreDoc[] hits = FloatPointNearestNeighbor.nearest(newSearcher(r, false), "point", 2, 45.0f, 50.0f).scoreDocs;
+ assertEquals("0", r.document(hits[0].doc).getField("id").stringValue());
+ assertEquals("1", r.document(hits[1].doc).getField("id").stringValue());
+
+ r.close();
+ w.close();
+ dir.close();
+ }
+
+ public void testNearestNeighborWithNoDocs() throws Exception {
+ Directory dir = newDirectory();
+ RandomIndexWriter w = new RandomIndexWriter(random(), dir, getIndexWriterConfig());
+ DirectoryReader r = w.getReader();
+ // can't wrap because we require Lucene60PointsFormat directly but e.g. ParallelReader wraps with its own points impl:
+ assertEquals(0, FloatPointNearestNeighbor.nearest(newSearcher(r, false), "point", 1, 40.0f, 50.0f).scoreDocs.length);
+ r.close();
+ w.close();
+ dir.close();
+ }
+
+ public void testNearestNeighborRandom() throws Exception {
+ Directory dir;
+ int numPoints = atLeast(5000);
+ if (numPoints > 100000) {
+ dir = newFSDirectory(createTempDir(getClass().getSimpleName()));
+ } else {
+ dir = newDirectory();
+ }
+ IndexWriterConfig iwc = getIndexWriterConfig();
+ iwc.setMergePolicy(newLogMergePolicy());
+ iwc.setMergeScheduler(new SerialMergeScheduler());
+ RandomIndexWriter w = new RandomIndexWriter(random(), dir, iwc);
+
+ int dims = TestUtil.nextInt(random(), 1, PointValues.MAX_DIMENSIONS);
+ float[][] values = new float[numPoints][dims];
+ for (int id = 0 ; id < numPoints ; ++id) {
+ for (int dim = 0 ; dim < dims ; ++dim) {
+ Float f = Float.NaN;
+ while (f.isNaN()) {
+ f = Float.intBitsToFloat(random().nextInt());
+ }
+ values[id][dim] = f;
+ }
+ Document doc = new Document();
+ doc.add(new FloatPoint("point", values[id]));
+ doc.add(new StoredField("id", id));
+ w.addDocument(doc);
+ }
+
+ if (random().nextBoolean()) {
+ w.forceMerge(1);
+ }
+
+ DirectoryReader r = w.getReader();
+ if (VERBOSE) {
+ System.out.println("TEST: reader=" + r);
+ }
+ // can't wrap because we require Lucene60PointsFormat directly but e.g. ParallelReader wraps with its own points impl:
+ IndexSearcher s = newSearcher(r, false);
+ int iters = atLeast(100);
+ for (int iter = 0 ; iter < iters ; ++iter) {
+ if (VERBOSE) {
+ System.out.println("\nTEST: iter=" + iter);
+ }
+ float[] origin = new float[dims];
+ for (int dim = 0 ; dim < dims ; ++dim) {
+ Float f = Float.NaN;
+ while (f.isNaN()) {
+ f = Float.intBitsToFloat(random().nextInt());
+ }
+ origin[dim] = f;
+ }
+
+ // dumb brute force search to get the expected result:
+ FloatPointNearestNeighbor.NearestHit[] expectedHits = new FloatPointNearestNeighbor.NearestHit[numPoints];
+ for (int id = 0 ; id < numPoints ; ++id) {
+ FloatPointNearestNeighbor.NearestHit hit = new FloatPointNearestNeighbor.NearestHit();
+ hit.distanceSquared = FloatPointNearestNeighbor.euclideanDistanceSquared(origin, values[id]);
+ hit.docID = id;
+ expectedHits[id] = hit;
+ }
+
+ Arrays.sort(expectedHits, (a, b) -> {
+ int cmp = Double.compare(a.distanceSquared, b.distanceSquared);
+ return cmp != 0 ? cmp : a.docID - b.docID; // tie break by smaller id
+ });
+
+ int topK = TestUtil.nextInt(random(), 1, numPoints);
+
+ if (VERBOSE) {
+ System.out.println("\nhits for origin=" + Arrays.toString(origin));
+ }
+
+ ScoreDoc[] hits = FloatPointNearestNeighbor.nearest(s, "point", topK, origin).scoreDocs;
+ assertEquals("fewer than expected hits: ", topK, hits.length);
+
+ if (VERBOSE) {
+ for (int i = 0 ; i < topK ; ++i) {
+ FloatPointNearestNeighbor.NearestHit expected = expectedHits[i];
+ FieldDoc actual = (FieldDoc)hits[i];
+ Document actualDoc = r.document(actual.doc);
+ System.out.println("hit " + i);
+ System.out.println(" expected id=" + expected.docID + " " + Arrays.toString(values[expected.docID])
+ + " distance=" + (float)Math.sqrt(expected.distanceSquared) + " distanceSquared=" + expected.distanceSquared);
+ System.out.println(" actual id=" + actualDoc.getField("id") + " distance=" + actual.fields[0]);
+ }
+ }
+
+ for (int i = 0 ; i < topK ; ++i) {
+ FloatPointNearestNeighbor.NearestHit expected = expectedHits[i];
+ FieldDoc actual = (FieldDoc)hits[i];
+ assertEquals("hit " + i + ":", expected.docID, actual.doc);
+ assertEquals("hit " + i + ":", (float)Math.sqrt(expected.distanceSquared), (Float)actual.fields[0], 0.000001);
+ }
+ }
+
+ r.close();
+ w.close();
+ dir.close();
+ }
+
+ private IndexWriterConfig getIndexWriterConfig() {
+ IndexWriterConfig iwc = newIndexWriterConfig();
+ iwc.setCodec(Codec.forName("Lucene70"));
+ return iwc;
+ }
+}
[24/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11414: Add
gammaDistribution Stream Evaluator
Posted by no...@apache.org.
SOLR-11414: Add gammaDistribution Stream Evaluator
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/3f94f2ed
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/3f94f2ed
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/3f94f2ed
Branch: refs/heads/feature/autoscaling
Commit: 3f94f2ed4cd3e0126a588952946855337c940959
Parents: 790533d
Author: Joel Bernstein <jb...@apache.org>
Authored: Thu Sep 28 14:08:38 2017 -0400
Committer: Joel Bernstein <jb...@apache.org>
Committed: Thu Sep 28 14:08:38 2017 -0400
----------------------------------------------------------------------
.../org/apache/solr/handler/StreamHandler.java | 2 +-
.../io/eval/GammaDistributionEvaluator.java | 48 ++++++++++++++++++
.../solrj/io/stream/StreamExpressionTest.java | 51 ++++++++++++++++++++
3 files changed, 100 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/3f94f2ed/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
index 559ccea..127c44a 100644
--- a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
@@ -306,7 +306,7 @@ public class StreamHandler extends RequestHandlerBase implements SolrCoreAware,
.withFunctionName("mode", ModeEvaluator.class)
.withFunctionName("logNormalDistribution", LogNormalDistributionEvaluator.class)
.withFunctionName("zipFDistribution", ZipFDistributionEvaluator.class)
-
+ .withFunctionName("gammaDistribution", GammaDistributionEvaluator.class)
// Boolean Stream Evaluators
.withFunctionName("and", AndEvaluator.class)
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/3f94f2ed/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/GammaDistributionEvaluator.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/GammaDistributionEvaluator.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/GammaDistributionEvaluator.java
new file mode 100644
index 0000000..52edc6e
--- /dev/null
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/GammaDistributionEvaluator.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.io.eval;
+
+import java.io.IOException;
+import java.util.Locale;
+
+import org.apache.commons.math3.distribution.GammaDistribution;
+import org.apache.solr.client.solrj.io.stream.expr.StreamExpression;
+import org.apache.solr.client.solrj.io.stream.expr.StreamFactory;
+
+public class GammaDistributionEvaluator extends RecursiveNumericEvaluator implements TwoValueWorker {
+
+ private static final long serialVersionUID = 1;
+
+ public GammaDistributionEvaluator(StreamExpression expression, StreamFactory factory) throws IOException {
+ super(expression, factory);
+ }
+
+ @Override
+ public Object doWork(Object first, Object second) throws IOException{
+ if(null == first){
+ throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - null found for the first value",toExpression(constructingFactory)));
+ }
+ if(null == second){
+ throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - null found for the second value",toExpression(constructingFactory)));
+ }
+
+ Number shape = (Number)first;
+ Number scale = (Number)second;
+
+ return new GammaDistribution(shape.doubleValue(), scale.doubleValue());
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/3f94f2ed/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
index 1326740..6799371 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
@@ -6398,6 +6398,8 @@ public class StreamExpressionTest extends SolrCloudTestCase {
}
}
+
+
@Test
public void testEnumeratedDistribution() throws Exception {
String cexpr = "let(a=uniformIntegerDistribution(1, 10)," +
@@ -6625,6 +6627,55 @@ public class StreamExpressionTest extends SolrCloudTestCase {
assertTrue(f.doubleValue() < g.doubleValue());
}
+
+ @Test
+ public void testGammaDistribution() throws Exception {
+ String cexpr = "let(echo=true, " +
+ "a=describe(sample(gammaDistribution(1, 10),10000)), " +
+ "b=describe(sample(gammaDistribution(3, 10),10000)), " +
+ "c=describe(sample(gammaDistribution(5, 10),10000))," +
+ "d=describe(sample(gammaDistribution(7, 10),10000))," +
+ "e=mean(sample(gammaDistribution(1, 10),10000))," +
+ "f=mean(sample(gammaDistribution(1, 20),10000))," +
+ "g=mean(sample(gammaDistribution(1, 30),10000)))";
+
+ ModifiableSolrParams paramsLoc = new ModifiableSolrParams();
+ paramsLoc.set("expr", cexpr);
+ paramsLoc.set("qt", "/stream");
+ String url = cluster.getJettySolrRunners().get(0).getBaseUrl().toString()+"/"+COLLECTIONORALIAS;
+ TupleStream solrStream = new SolrStream(url, paramsLoc);
+ StreamContext context = new StreamContext();
+ solrStream.setStreamContext(context);
+ List<Tuple> tuples = getTuples(solrStream);
+ assertTrue(tuples.size() == 1);
+ Map a = (Map)tuples.get(0).get("a");
+ Map b = (Map)tuples.get(0).get("b");
+ Map c = (Map)tuples.get(0).get("c");
+ Map d = (Map)tuples.get(0).get("d");
+
+ Number sa = (Number)a.get("skewness");
+ Number sb = (Number)b.get("skewness");
+ Number sc = (Number)c.get("skewness");
+ Number sd = (Number)d.get("skewness");
+
+ //Test shape change
+ assertTrue(sa.doubleValue() > sb.doubleValue());
+ assertTrue(sb.doubleValue() > sc.doubleValue());
+ assertTrue(sc.doubleValue() > sd.doubleValue());
+
+ //Test scale change
+
+ Number e = (Number)tuples.get(0).get("e");
+ Number f = (Number)tuples.get(0).get("f");
+ Number g = (Number)tuples.get(0).get("g");
+
+ assertTrue(e.doubleValue() < f.doubleValue());
+ assertTrue(f.doubleValue() < g.doubleValue());
+ }
+
+
+
+
@Test
public void testLogNormalDistribution() throws Exception {
String cexpr = "let(echo=true, " +
[42/50] [abbrv] lucene-solr:feature/autoscaling: LUCENE-7974: make
NearestHit public, and add javadocs, to make precommit happy
Posted by no...@apache.org.
LUCENE-7974: make NearestHit public, and add javadocs, to make precommit happy
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/73f34033
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/73f34033
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/73f34033
Branch: refs/heads/feature/autoscaling
Commit: 73f3403381b95a908543cf859b2f43f28cb9a34a
Parents: 0062e99
Author: Mike McCandless <mi...@apache.org>
Authored: Mon Oct 2 09:42:17 2017 -0400
Committer: Mike McCandless <mi...@apache.org>
Committed: Mon Oct 2 09:42:17 2017 -0400
----------------------------------------------------------------------
.../org/apache/lucene/document/FloatPointNearestNeighbor.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/73f34033/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
----------------------------------------------------------------------
diff --git a/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java b/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
index d3360a8..b7859d0 100644
--- a/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
+++ b/lucene/sandbox/src/java/org/apache/lucene/document/FloatPointNearestNeighbor.java
@@ -193,7 +193,8 @@ public class FloatPointNearestNeighbor {
}
}
- static class NearestHit {
+ /** Holds one hit from {@link FloatPointNearestNeighbor#nearest} */
+ public static class NearestHit {
public int docID;
public double distanceSquared;
[20/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11400: Add
logNormalDistribution Stream Evaluator
Posted by no...@apache.org.
SOLR-11400: Add logNormalDistribution Stream Evaluator
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/6b3447de
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/6b3447de
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/6b3447de
Branch: refs/heads/feature/autoscaling
Commit: 6b3447defd54783986edf4f7f818d7a6444e85f4
Parents: cc9c867
Author: Joel Bernstein <jb...@apache.org>
Authored: Wed Sep 27 13:52:35 2017 -0400
Committer: Joel Bernstein <jb...@apache.org>
Committed: Wed Sep 27 13:53:03 2017 -0400
----------------------------------------------------------------------
.../org/apache/solr/handler/StreamHandler.java | 3 +
.../io/eval/LogNormalDistributionEvaluator.java | 48 ++++++++++++
.../client/solrj/io/eval/ModeEvaluator.java | 64 ++++++++++++++++
.../solrj/io/stream/StreamExpressionTest.java | 77 +++++++++++++++++---
4 files changed, 180 insertions(+), 12 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/6b3447de/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
index 4f50ca9..4810759 100644
--- a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
@@ -29,6 +29,7 @@ import java.util.Iterator;
import java.util.List;
import java.util.Map;
+import org.apache.commons.math3.distribution.LogNormalDistribution;
import org.apache.solr.client.solrj.io.ModelCache;
import org.apache.solr.client.solrj.io.SolrClientCache;
import org.apache.solr.client.solrj.io.Tuple;
@@ -303,6 +304,8 @@ public class StreamHandler extends RequestHandlerBase implements SolrCoreAware,
.withFunctionName("constantDistribution", ConstantDistributionEvaluator.class)
.withFunctionName("weibullDistribution", WeibullDistributionEvaluator.class)
.withFunctionName("mean", MeanEvaluator.class)
+ .withFunctionName("mode", ModeEvaluator.class)
+ .withFunctionName("logNormalDistribution", LogNormalDistributionEvaluator.class)
// Boolean Stream Evaluators
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/6b3447de/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/LogNormalDistributionEvaluator.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/LogNormalDistributionEvaluator.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/LogNormalDistributionEvaluator.java
new file mode 100644
index 0000000..6b15bee
--- /dev/null
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/LogNormalDistributionEvaluator.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.io.eval;
+
+import java.io.IOException;
+import java.util.Locale;
+
+import org.apache.commons.math3.distribution.LogNormalDistribution;
+import org.apache.solr.client.solrj.io.stream.expr.StreamExpression;
+import org.apache.solr.client.solrj.io.stream.expr.StreamFactory;
+
+public class LogNormalDistributionEvaluator extends RecursiveNumericEvaluator implements TwoValueWorker {
+
+ private static final long serialVersionUID = 1;
+
+ public LogNormalDistributionEvaluator(StreamExpression expression, StreamFactory factory) throws IOException {
+ super(expression, factory);
+ }
+
+ @Override
+ public Object doWork(Object first, Object second) throws IOException{
+ if(null == first){
+ throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - null found for the first value",toExpression(constructingFactory)));
+ }
+ if(null == second){
+ throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - null found for the second value",toExpression(constructingFactory)));
+ }
+
+ Number shape = (Number)first;
+ Number scale = (Number)second;
+
+ return new LogNormalDistribution(scale.doubleValue(), shape.doubleValue());
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/6b3447de/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/ModeEvaluator.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/ModeEvaluator.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/ModeEvaluator.java
new file mode 100644
index 0000000..b2529c6
--- /dev/null
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/ModeEvaluator.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.io.eval;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+
+import org.apache.solr.client.solrj.io.stream.expr.StreamExpression;
+import org.apache.solr.client.solrj.io.stream.expr.StreamFactory;
+import org.apache.commons.math3.stat.StatUtils;
+
+
+public class ModeEvaluator extends RecursiveObjectEvaluator implements OneValueWorker {
+ protected static final long serialVersionUID = 1L;
+
+ public ModeEvaluator(StreamExpression expression, StreamFactory factory) throws IOException{
+ super(expression, factory);
+
+ if(1 != containedEvaluators.size()){
+ throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - expecting exactly 1 value but found %d",expression,containedEvaluators.size()));
+ }
+ }
+
+ @Override
+ public Object doWork(Object value) throws IOException{
+ if(null == value){
+ throw new IOException(String.format(Locale.ROOT, "Unable to find %s(...) because the value is null", constructingFactory.getFunctionName(getClass())));
+ }
+ else if(value instanceof List){
+ List<Number> c = (List<Number>) value;
+ double[] data = new double[c.size()];
+ for(int i=0; i< c.size(); i++) {
+ data[i] = c.get(i).doubleValue();
+ }
+
+ double[] mode = StatUtils.mode(data);
+ List<Number> l = new ArrayList();
+ for(double d : mode) {
+ l.add(d);
+ }
+
+ return l;
+ }
+ else{
+ throw new IOException(String.format(Locale.ROOT, "Unable to find %s(...) because the value is not a collection, instead a %s was found", constructingFactory.getFunctionName(getClass()), value.getClass().getSimpleName()));
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/6b3447de/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
index 44b8e54..318f8f6 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
@@ -6555,11 +6555,13 @@ public class StreamExpressionTest extends SolrCloudTestCase {
@Test
public void testWeibullDistribution() throws Exception {
String cexpr = "let(echo=true, " +
- "a=percentile(sample(weibullDistribution(1, 10),5000), 50), " +
- "b=percentile(sample(weibullDistribution(1, 50),5000), 50), " +
- "c=percentile(sample(weibullDistribution(1, 100),5000), 50)," +
- "d=percentile(sample(weibullDistribution(4, 10),5000), 50)," +
- "e=percentile(sample(weibullDistribution(8, 10),5000), 50))";
+ "a=describe(sample(weibullDistribution(.1, 10),10000)), " +
+ "b=describe(sample(weibullDistribution(.5, 10),10000)), " +
+ "c=describe(sample(weibullDistribution(1, 10),10000))," +
+ "d=describe(sample(weibullDistribution(6, 10),10000))," +
+ "e=mean(sample(weibullDistribution(1, 10),10000))," +
+ "f=mean(sample(weibullDistribution(1, 20),10000))," +
+ "g=mean(sample(weibullDistribution(1, 30),10000)))";
ModifiableSolrParams paramsLoc = new ModifiableSolrParams();
paramsLoc.set("expr", cexpr);
@@ -6570,17 +6572,68 @@ public class StreamExpressionTest extends SolrCloudTestCase {
solrStream.setStreamContext(context);
List<Tuple> tuples = getTuples(solrStream);
assertTrue(tuples.size() == 1);
- Number a = (Number)tuples.get(0).get("a");
- Number b = (Number)tuples.get(0).get("b");
- Number c = (Number)tuples.get(0).get("c");
+ Map a = (Map)tuples.get(0).get("a");
+ Map b = (Map)tuples.get(0).get("b");
+ Map c = (Map)tuples.get(0).get("c");
+ Map d = (Map)tuples.get(0).get("d");
+
+ Number sa = (Number)a.get("skewness");
+ Number sb = (Number)b.get("skewness");
+ Number sc = (Number)c.get("skewness");
+ Number sd = (Number)d.get("skewness");
+
+ //Test shape change
+ assertTrue(sa.doubleValue() > sb.doubleValue());
+ assertTrue(sb.doubleValue() > sc.doubleValue());
+ assertTrue(sc.doubleValue() > sd.doubleValue());
+ assertTrue(sd.doubleValue() < 0.0);
+
+ //Test scale change
+
+ Number e = (Number)tuples.get(0).get("e");
+ Number f = (Number)tuples.get(0).get("f");
+ Number g = (Number)tuples.get(0).get("g");
+
+ assertTrue(e.doubleValue() < f.doubleValue());
+ assertTrue(f.doubleValue() < g.doubleValue());
+ }
+
+ @Test
+ public void testLogNormalDistribution() throws Exception {
+ String cexpr = "let(echo=true, " +
+ "a=describe(sample(logNormalDistribution(.1, 0),10000)), " +
+ "b=describe(sample(logNormalDistribution(.3, 0),10000)), " +
+ "c=describe(sample(logNormalDistribution(.6, 0),10000))," +
+ "d=mean(sample(logNormalDistribution(.3, 0),10000)), " +
+ "e=mean(sample(logNormalDistribution(.3, 2),10000)), " +
+ ")";
+
+ ModifiableSolrParams paramsLoc = new ModifiableSolrParams();
+ paramsLoc.set("expr", cexpr);
+ paramsLoc.set("qt", "/stream");
+ String url = cluster.getJettySolrRunners().get(0).getBaseUrl().toString()+"/"+COLLECTIONORALIAS;
+ TupleStream solrStream = new SolrStream(url, paramsLoc);
+ StreamContext context = new StreamContext();
+ solrStream.setStreamContext(context);
+ List<Tuple> tuples = getTuples(solrStream);
+ assertTrue(tuples.size() == 1);
+ Map a = (Map)tuples.get(0).get("a");
+ Map b = (Map)tuples.get(0).get("b");
+ Map c = (Map)tuples.get(0).get("c");
+
+ Number sa = (Number)a.get("skewness");
+ Number sb = (Number)b.get("skewness");
+ Number sc = (Number)c.get("skewness");
+
+ assertTrue(sa.doubleValue() < sb.doubleValue());
+ assertTrue(sb.doubleValue() < sc.doubleValue());
+
Number d = (Number)tuples.get(0).get("d");
Number e = (Number)tuples.get(0).get("e");
- assertTrue(a.doubleValue() < b.doubleValue());
- assertTrue(b.doubleValue() < c.doubleValue());
- assertTrue(a.doubleValue() < d.doubleValue());
+
assertTrue(d.doubleValue() < e.doubleValue());
- }
+ }
@Test
public void testMean() throws Exception {
[18/50] [abbrv] lucene-solr:feature/autoscaling: Improve Readme with
ivy-bootstrap and Ivy known workaround
Posted by no...@apache.org.
Improve Readme with ivy-bootstrap and Ivy known workaround
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/e1485749
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/e1485749
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/e1485749
Branch: refs/heads/feature/autoscaling
Commit: e148574997d56e0db03d68de4f3525be41a8afed
Parents: ee687c39
Author: antonmry <an...@galiglobal.com>
Authored: Wed Sep 27 14:04:00 2017 +0200
Committer: antonmry <an...@galiglobal.com>
Committed: Wed Sep 27 15:58:20 2017 +0200
----------------------------------------------------------------------
README.md | 6 ++++++
1 file changed, 6 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/e1485749/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 7181297..50da250 100644
--- a/README.md
+++ b/README.md
@@ -41,6 +41,12 @@ Lucene and Solr, run:
`ant compile`
+If you see an error about Ivy missing when invoking ant (e.g., `.ant/lib does not exist`), run `ant ivy-bootstrap` and retry.
+
+Sometimes you may face issues with Ivy (e.g., instance a wrongly downloaded artifact). Cleaning up the Ivy cache and retrying is a workaround for most of such issues:
+
+`rm -rf ~/.ivy2/cache`
+
The Solr server can then be packaged and prepared for startup by running the
following command from the `solr/` directory:
[12/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11399:
hl.fragsize was ignored when hl.bs.type=SEPARATOR
Posted by no...@apache.org.
SOLR-11399: hl.fragsize was ignored when hl.bs.type=SEPARATOR
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/3b0f46b2
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/3b0f46b2
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/3b0f46b2
Branch: refs/heads/feature/autoscaling
Commit: 3b0f46b27defeb0bdcf70bc647872742a952822a
Parents: 6fe7829
Author: David Smiley <ds...@apache.org>
Authored: Tue Sep 26 10:32:00 2017 -0400
Committer: David Smiley <ds...@apache.org>
Committed: Tue Sep 26 10:32:00 2017 -0400
----------------------------------------------------------------------
solr/CHANGES.txt | 3 +++
.../solr/highlight/UnifiedSolrHighlighter.java | 18 +++++++++++-------
.../highlight/TestUnifiedSolrHighlighter.java | 3 +++
3 files changed, 17 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/3b0f46b2/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 18219d4..2e1c70d 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -148,6 +148,9 @@ Bug Fixes
* SOLR-11297: Message "Lock held by this virtual machine" during startup. Solr is trying to start some cores twice.
(Luiz Armesto, Shawn Heisey, Erick Erickson)
+* SOLR-11399: The UnifiedHighlighter was ignoring the hl.fragsize parameter when hl.bs.type=SEPARATOR
+ (Marc Morissette via David Smiley)
+
Optimizations
----------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/3b0f46b2/solr/core/src/java/org/apache/solr/highlight/UnifiedSolrHighlighter.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/highlight/UnifiedSolrHighlighter.java b/solr/core/src/java/org/apache/solr/highlight/UnifiedSolrHighlighter.java
index 48cb874..a4beaba 100644
--- a/solr/core/src/java/org/apache/solr/highlight/UnifiedSolrHighlighter.java
+++ b/solr/core/src/java/org/apache/solr/highlight/UnifiedSolrHighlighter.java
@@ -310,15 +310,19 @@ public class UnifiedSolrHighlighter extends SolrHighlighter implements PluginInf
String type = params.getFieldParam(field, HighlightParams.BS_TYPE);
if (fragsize == 0 || "WHOLE".equals(type)) { // 0 is special value; no fragmenting
return new WholeBreakIterator();
- } else if ("SEPARATOR".equals(type)) {
+ }
+
+ BreakIterator baseBI;
+ if ("SEPARATOR".equals(type)) {
char customSep = parseBiSepChar(params.getFieldParam(field, HighlightParams.BS_SEP));
- return new CustomSeparatorBreakIterator(customSep);
+ baseBI = new CustomSeparatorBreakIterator(customSep);
+ } else {
+ String language = params.getFieldParam(field, HighlightParams.BS_LANGUAGE);
+ String country = params.getFieldParam(field, HighlightParams.BS_COUNTRY);
+ String variant = params.getFieldParam(field, HighlightParams.BS_VARIANT);
+ Locale locale = parseLocale(language, country, variant);
+ baseBI = parseBreakIterator(type, locale);
}
- String language = params.getFieldParam(field, HighlightParams.BS_LANGUAGE);
- String country = params.getFieldParam(field, HighlightParams.BS_COUNTRY);
- String variant = params.getFieldParam(field, HighlightParams.BS_VARIANT);
- Locale locale = parseLocale(language, country, variant);
- BreakIterator baseBI = parseBreakIterator(type, locale);
if (fragsize <= 1) { // no real minimum size
return baseBI;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/3b0f46b2/solr/core/src/test/org/apache/solr/highlight/TestUnifiedSolrHighlighter.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/highlight/TestUnifiedSolrHighlighter.java b/solr/core/src/test/org/apache/solr/highlight/TestUnifiedSolrHighlighter.java
index ad1ca03..aeffed1 100644
--- a/solr/core/src/test/org/apache/solr/highlight/TestUnifiedSolrHighlighter.java
+++ b/solr/core/src/test/org/apache/solr/highlight/TestUnifiedSolrHighlighter.java
@@ -256,6 +256,9 @@ public class TestUnifiedSolrHighlighter extends SolrTestCaseJ4 {
req("q", "text:document", "sort", "id asc", "hl", "true", "hl.bs.type", "SEPARATOR","hl.bs.separator","#","hl.fragsize", "-1"),
"//lst[@name='highlighting']/lst[@name='104']/arr[@name='text']/str='While the other <em>document</em> contains the same #'");
+ assertQ("CUSTOM breakiterator with fragsize 70",
+ req("q", "text:document", "sort", "id asc", "hl", "true", "hl.bs.type", "SEPARATOR","hl.bs.separator","#","hl.fragsize", "70"),
+ "//lst[@name='highlighting']/lst[@name='103']/arr[@name='text']/str='This <em>document</em> contains # special characters, while the other <em>document</em> contains the same #'");
}
public void testFragsize() {
[03/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11382: Maven
build: Build the test-jar for lucene-spatial3d,
which lucene-spatial tests now depend on
Posted by no...@apache.org.
SOLR-11382: Maven build: Build the test-jar for lucene-spatial3d, which lucene-spatial tests now depend on
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/001fa289
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/001fa289
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/001fa289
Branch: refs/heads/feature/autoscaling
Commit: 001fa289e446e45e169a3fbd680ed63e393f3447
Parents: 6391a75
Author: Steve Rowe <sa...@gmail.com>
Authored: Mon Sep 25 20:09:27 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Mon Sep 25 20:09:27 2017 -0400
----------------------------------------------------------------------
dev-tools/maven/lucene/spatial3d/pom.xml.template | 13 +++++++++++++
1 file changed, 13 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/001fa289/dev-tools/maven/lucene/spatial3d/pom.xml.template
----------------------------------------------------------------------
diff --git a/dev-tools/maven/lucene/spatial3d/pom.xml.template b/dev-tools/maven/lucene/spatial3d/pom.xml.template
index 27d3bb8..342ed41 100644
--- a/dev-tools/maven/lucene/spatial3d/pom.xml.template
+++ b/dev-tools/maven/lucene/spatial3d/pom.xml.template
@@ -53,5 +53,18 @@
<build>
<sourceDirectory>${module-path}/src/java</sourceDirectory>
<testSourceDirectory>${module-path}/src/test</testSourceDirectory>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-jar-plugin</artifactId>
+ <executions>
+ <execution>
+ <goals>
+ <goal>test-jar</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
</build>
</project>
[09/50] [abbrv] lucene-solr:feature/autoscaling: LUCENE-7970: Handle
the case where we generate only a pair of identical planes,
as might happen when the center is near a pole.
Posted by no...@apache.org.
LUCENE-7970: Handle the case where we generate only a pair of identical planes, as might happen when the center is near a pole.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/49583ed1
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/49583ed1
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/49583ed1
Branch: refs/heads/feature/autoscaling
Commit: 49583ed13da97cd11c8b9d1b3a88079175a6c69d
Parents: f8f1956
Author: Karl Wright <Da...@gmail.com>
Authored: Tue Sep 26 03:20:32 2017 -0400
Committer: Karl Wright <Da...@gmail.com>
Committed: Tue Sep 26 03:20:32 2017 -0400
----------------------------------------------------------------------
.../lucene/spatial3d/geom/GeoExactCircle.java | 30 +++++++++++++-------
1 file changed, 20 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/49583ed1/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoExactCircle.java
----------------------------------------------------------------------
diff --git a/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoExactCircle.java b/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoExactCircle.java
index b6ae081..03eb5ce 100644
--- a/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoExactCircle.java
+++ b/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoExactCircle.java
@@ -119,8 +119,9 @@ class GeoExactCircle extends GeoBaseCircle {
this.notableEdgePoints = null;
} else {
this.circlePlanes = new ArrayList<>();
- this.notableEdgePoints = new ArrayList<>();
- this.eitherBounds = new HashMap<>();
+ // If it turns out that there's only one circle plane, this array will be populated but unused
+ final List<GeoPoint[]> notableEdgePoints = new ArrayList<>();
+
// We construct approximation planes until we have a low enough error estimate
final List<ApproximationSlice> slices = new ArrayList<>(100);
// Construct four cardinal points, and then we'll build the first two planes
@@ -154,9 +155,11 @@ class GeoExactCircle extends GeoBaseCircle {
final GeoPoint interpPoint2 = planetModel.surfacePointOnBearing(center, cutoffAngle, interpPoint2Bearing);
// Is this point on the plane? (that is, is the approximation good enough?)
if (Math.abs(thisSlice.plane.evaluate(interpPoint1)) < actualAccuracy && Math.abs(thisSlice.plane.evaluate(interpPoint2)) < actualAccuracy) {
- // Good enough; add it to the list of planes
- circlePlanes.add(thisSlice.plane);
- notableEdgePoints.add(new GeoPoint[]{thisSlice.endPoint1, thisSlice.endPoint2});
+ // Good enough; add it to the list of planes, unless it was identical to the previous plane
+ if (circlePlanes.size() == 0 || !circlePlanes.get(circlePlanes.size()-1).isNumericallyIdentical(thisSlice.plane)) {
+ circlePlanes.add(thisSlice.plane);
+ notableEdgePoints.add(new GeoPoint[]{thisSlice.endPoint1, thisSlice.endPoint2});
+ }
} else {
// Split the plane into two, and add it back to the end
slices.add(new ApproximationSlice(center,
@@ -173,11 +176,18 @@ class GeoExactCircle extends GeoBaseCircle {
//System.out.println("Number of planes needed: "+circlePlanes.size());
// Compute bounds
- for (int i = 0; i < circlePlanes.size(); i++) {
- final SidedPlane thisPlane = circlePlanes.get(i);
- final SidedPlane previousPlane = (i == 0)?circlePlanes.get(circlePlanes.size()-1):circlePlanes.get(i-1);
- final SidedPlane nextPlane = (i == circlePlanes.size()-1)?circlePlanes.get(0):circlePlanes.get(i+1);
- eitherBounds.put(thisPlane, new EitherBound(previousPlane, nextPlane));
+ if (circlePlanes.size() == 1) {
+ this.eitherBounds = null;
+ this.notableEdgePoints = null;
+ } else {
+ this.notableEdgePoints = notableEdgePoints;
+ this.eitherBounds = new HashMap<>(circlePlanes.size());
+ for (int i = 0; i < circlePlanes.size(); i++) {
+ final SidedPlane thisPlane = circlePlanes.get(i);
+ final SidedPlane previousPlane = (i == 0)?circlePlanes.get(circlePlanes.size()-1):circlePlanes.get(i-1);
+ final SidedPlane nextPlane = (i == circlePlanes.size()-1)?circlePlanes.get(0):circlePlanes.get(i+1);
+ eitherBounds.put(thisPlane, new EitherBound(previousPlane, nextPlane));
+ }
}
}
}
[31/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11420: sql
Streaming Expression should use the default collection if none is provided
Posted by no...@apache.org.
SOLR-11420: sql Streaming Expression should use the default collection if none is provided
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/71eb59e0
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/71eb59e0
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/71eb59e0
Branch: refs/heads/feature/autoscaling
Commit: 71eb59e0435f744db2d5cf8c58b1fca70af9b824
Parents: 633af79
Author: Joel Bernstein <jb...@apache.org>
Authored: Fri Sep 29 10:36:26 2017 -0400
Committer: Joel Bernstein <jb...@apache.org>
Committed: Fri Sep 29 10:36:26 2017 -0400
----------------------------------------------------------------------
.../org/apache/solr/client/solrj/io/stream/SqlStream.java | 7 +------
.../solr/client/solrj/io/stream/expr/StreamFactory.java | 6 ++++++
.../solr/client/solrj/io/stream/StreamExpressionTest.java | 10 ++++++++++
3 files changed, 17 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/71eb59e0/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/SqlStream.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/SqlStream.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/SqlStream.java
index d2296b7..93345e7 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/SqlStream.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/SqlStream.java
@@ -73,12 +73,7 @@ public class SqlStream extends TupleStream implements Expressible {
// Collection Name
if(null == collectionName){
- throw new IOException(String.format(Locale.ROOT,"invalid expression %s - collectionName expected as first operand",expression));
- }
-
- // Validate there are no unknown parameters - zkHost and alias are namedParameter so we don't need to count it twice
- if(expression.getParameters().size() != 1 + namedParams.size()){
- throw new IOException(String.format(Locale.ROOT,"invalid expression %s - unknown operands found",expression));
+ collectionName = factory.getDefaultCollection();
}
// Named parameters - passed directly to solr as solrparams
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/71eb59e0/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamFactory.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamFactory.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamFactory.java
index f4c9b58..321545b 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamFactory.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamFactory.java
@@ -46,6 +46,7 @@ public class StreamFactory implements Serializable {
private transient HashMap<String,String> collectionZkHosts;
private transient HashMap<String,Class<? extends Expressible>> functionNames;
private transient String defaultZkHost;
+ private transient String defaultCollection;
public StreamFactory(){
collectionZkHosts = new HashMap<>();
@@ -54,9 +55,14 @@ public class StreamFactory implements Serializable {
public StreamFactory withCollectionZkHost(String collectionName, String zkHost){
this.collectionZkHosts.put(collectionName, zkHost);
+ this.defaultCollection = collectionName;
return this;
}
+ public String getDefaultCollection() {
+ return defaultCollection;
+ }
+
public StreamFactory withDefaultZkHost(String zkHost) {
this.defaultZkHost = zkHost;
return this;
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/71eb59e0/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
index 949d224..d84120f 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
@@ -279,6 +279,16 @@ public class StreamExpressionTest extends SolrCloudTestCase {
assert (tuples.size() == 5);
assertOrder(tuples, 0, 1, 2, 3, 4);
+ //Test with using the default collection
+ solrParams = new ModifiableSolrParams();
+ solrParams.add("qt", "/stream");
+ solrParams.add("expr", "sql(stmt=\"select id from collection1 order by a_i asc\")");
+ solrStream = new SolrStream(shardUrls.get(0), solrParams);
+ solrStream.setStreamContext(streamContext);
+ tuples = getTuples(solrStream);
+ assert (tuples.size() == 5);
+ assertOrder(tuples, 0, 1, 2, 3, 4);
+
} finally {
solrClientCache.close();
}
[30/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11415: Add
betaDistribution Stream Evaluator
Posted by no...@apache.org.
SOLR-11415: Add betaDistribution Stream Evaluator
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/633af794
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/633af794
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/633af794
Branch: refs/heads/feature/autoscaling
Commit: 633af79406566b2323f589c83640f13b196a0312
Parents: 97f246b
Author: Joel Bernstein <jb...@apache.org>
Authored: Fri Sep 29 09:57:23 2017 -0400
Committer: Joel Bernstein <jb...@apache.org>
Committed: Fri Sep 29 09:57:45 2017 -0400
----------------------------------------------------------------------
.../org/apache/solr/handler/StreamHandler.java | 2 +
.../io/eval/BetaDistributionEvaluator.java | 48 ++++++++++++++++++++
.../solrj/io/stream/StreamExpressionTest.java | 45 ++++++++++++++++++
3 files changed, 95 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/633af794/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
index 127c44a..16a31fb 100644
--- a/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
+++ b/solr/core/src/java/org/apache/solr/handler/StreamHandler.java
@@ -307,6 +307,8 @@ public class StreamHandler extends RequestHandlerBase implements SolrCoreAware,
.withFunctionName("logNormalDistribution", LogNormalDistributionEvaluator.class)
.withFunctionName("zipFDistribution", ZipFDistributionEvaluator.class)
.withFunctionName("gammaDistribution", GammaDistributionEvaluator.class)
+ .withFunctionName("betaDistribution", BetaDistributionEvaluator.class)
+
// Boolean Stream Evaluators
.withFunctionName("and", AndEvaluator.class)
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/633af794/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/BetaDistributionEvaluator.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/BetaDistributionEvaluator.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/BetaDistributionEvaluator.java
new file mode 100644
index 0000000..288ecb9
--- /dev/null
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/BetaDistributionEvaluator.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.client.solrj.io.eval;
+
+import java.io.IOException;
+import java.util.Locale;
+
+import org.apache.commons.math3.distribution.BetaDistribution;
+import org.apache.solr.client.solrj.io.stream.expr.StreamExpression;
+import org.apache.solr.client.solrj.io.stream.expr.StreamFactory;
+
+public class BetaDistributionEvaluator extends RecursiveNumericEvaluator implements TwoValueWorker {
+
+ private static final long serialVersionUID = 1;
+
+ public BetaDistributionEvaluator(StreamExpression expression, StreamFactory factory) throws IOException {
+ super(expression, factory);
+ }
+
+ @Override
+ public Object doWork(Object first, Object second) throws IOException{
+ if(null == first){
+ throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - null found for the first value",toExpression(constructingFactory)));
+ }
+ if(null == second){
+ throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - null found for the second value",toExpression(constructingFactory)));
+ }
+
+ Number shape1 = (Number)first;
+ Number shape2 = (Number)second;
+
+ return new BetaDistribution(shape1.doubleValue(), shape2.doubleValue());
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/633af794/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
index 6f11382..949d224 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
@@ -6399,6 +6399,51 @@ public class StreamExpressionTest extends SolrCloudTestCase {
}
+ @Test
+ public void testBetaDistribution() throws Exception {
+ String cexpr = "let(a=sample(betaDistribution(1, 5), 50000), b=hist(a, 11), c=col(b, N))";
+ ModifiableSolrParams paramsLoc = new ModifiableSolrParams();
+ paramsLoc.set("expr", cexpr);
+ paramsLoc.set("qt", "/stream");
+ String url = cluster.getJettySolrRunners().get(0).getBaseUrl().toString()+"/"+COLLECTIONORALIAS;
+ TupleStream solrStream = new SolrStream(url, paramsLoc);
+ StreamContext context = new StreamContext();
+ solrStream.setStreamContext(context);
+ List<Tuple> tuples = getTuples(solrStream);
+ assertTrue(tuples.size() == 1);
+ List<Number> counts = (List<Number>)tuples.get(0).get("c");
+
+ int lastCount = Integer.MAX_VALUE;
+ for(Number number : counts) {
+ int current = number.intValue();
+ if(current > lastCount) {
+ throw new Exception("This beta distribution should be descending");
+ } else {
+ lastCount = current;
+ }
+ }
+
+ cexpr = "let(a=sample(betaDistribution(5, 1), 50000), b=hist(a, 11), c=col(b, N))";
+ paramsLoc = new ModifiableSolrParams();
+ paramsLoc.set("expr", cexpr);
+ paramsLoc.set("qt", "/stream");
+ solrStream = new SolrStream(url, paramsLoc);
+ context = new StreamContext();
+ solrStream.setStreamContext(context);
+ tuples = getTuples(solrStream);
+ assertTrue(tuples.size() == 1);
+ counts = (List<Number>)tuples.get(0).get("c");
+
+ lastCount = Integer.MIN_VALUE;
+ for(Number number : counts) {
+ int current = number.intValue();
+ if(current < lastCount) {
+ throw new Exception("This beta distribution should be ascending");
+ } else {
+ lastCount = current;
+ }
+ }
+ }
@Test
public void testEnumeratedDistribution() throws Exception {
[47/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-10285: Skip
LEADER messages when there are leader only shards
Posted by no...@apache.org.
SOLR-10285: Skip LEADER messages when there are leader only shards
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/fd2b4f3f
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/fd2b4f3f
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/fd2b4f3f
Branch: refs/heads/feature/autoscaling
Commit: fd2b4f3f868276a3b3e3b0b6845d1a7309c9cad0
Parents: 5b3a515
Author: Cao Manh Dat <da...@apache.org>
Authored: Tue Oct 3 09:00:02 2017 +0700
Committer: Cao Manh Dat <da...@apache.org>
Committed: Tue Oct 3 09:00:02 2017 +0700
----------------------------------------------------------------------
solr/CHANGES.txt | 2 +
.../org/apache/solr/cloud/ElectionContext.java | 38 ++++++---
.../solr/cloud/TestShardsWithSingleReplica.java | 84 ++++++++++++++++++++
3 files changed, 112 insertions(+), 12 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/fd2b4f3f/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 90ada02..177cdf7 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -171,6 +171,8 @@ Optimizations
* SOLR-8344: Decide default when requested fields are both column and row stored. (Cao Manh Dat, David Smiley)
+* SOLR-10285: Skip LEADER messages when there are leader only shards (Cao Manh Dat, Joshua Humphries)
+
Other Changes
----------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/fd2b4f3f/solr/core/src/java/org/apache/solr/cloud/ElectionContext.java
----------------------------------------------------------------------
diff --git a/solr/core/src/java/org/apache/solr/cloud/ElectionContext.java b/solr/core/src/java/org/apache/solr/cloud/ElectionContext.java
index 491ae00..ba88e75 100644
--- a/solr/core/src/java/org/apache/solr/cloud/ElectionContext.java
+++ b/solr/core/src/java/org/apache/solr/cloud/ElectionContext.java
@@ -115,6 +115,7 @@ class ShardLeaderElectionContextBase extends ElectionContext {
protected String shardId;
protected String collection;
protected LeaderElector leaderElector;
+ protected ZkStateReader zkStateReader;
private Integer leaderZkNodeParentVersion;
// Prevents a race between cancelling and becoming leader.
@@ -128,6 +129,7 @@ class ShardLeaderElectionContextBase extends ElectionContext {
collection, shardId), props, zkStateReader.getZkClient());
this.leaderElector = leaderElector;
this.zkClient = zkStateReader.getZkClient();
+ this.zkStateReader = zkStateReader;
this.shardId = shardId;
this.collection = collection;
}
@@ -216,14 +218,24 @@ class ShardLeaderElectionContextBase extends ElectionContext {
}
assert shardId != null;
- ZkNodeProps m = ZkNodeProps.fromKeyVals(Overseer.QUEUE_OPERATION,
- OverseerAction.LEADER.toLower(), ZkStateReader.SHARD_ID_PROP, shardId,
- ZkStateReader.COLLECTION_PROP, collection, ZkStateReader.BASE_URL_PROP,
- leaderProps.getProperties().get(ZkStateReader.BASE_URL_PROP),
- ZkStateReader.CORE_NAME_PROP,
- leaderProps.getProperties().get(ZkStateReader.CORE_NAME_PROP),
- ZkStateReader.STATE_PROP, Replica.State.ACTIVE.toString());
- Overseer.getStateUpdateQueue(zkClient).offer(Utils.toJSON(m));
+ boolean isAlreadyLeader = false;
+ if (zkStateReader.getClusterState() != null &&
+ zkStateReader.getClusterState().getCollection(collection).getSlice(shardId).getReplicas().size() < 2) {
+ Replica leader = zkStateReader.getLeader(collection, shardId);
+ if (leader != null
+ && leader.getBaseUrl().equals(leaderProps.get(ZkStateReader.BASE_URL_PROP))
+ && leader.getCoreName().equals(leaderProps.get(ZkStateReader.CORE_NAME_PROP))) {
+ isAlreadyLeader = true;
+ }
+ }
+ if (!isAlreadyLeader) {
+ ZkNodeProps m = ZkNodeProps.fromKeyVals(Overseer.QUEUE_OPERATION, OverseerAction.LEADER.toLower(),
+ ZkStateReader.SHARD_ID_PROP, shardId,
+ ZkStateReader.COLLECTION_PROP, collection,
+ ZkStateReader.BASE_URL_PROP, leaderProps.get(ZkStateReader.BASE_URL_PROP),
+ ZkStateReader.CORE_NAME_PROP, leaderProps.get(ZkStateReader.CORE_NAME_PROP));
+ Overseer.getStateUpdateQueue(zkClient).offer(Utils.toJSON(m));
+ }
}
public LeaderElector getLeaderElector() {
@@ -309,10 +321,12 @@ final class ShardLeaderElectionContext extends ShardLeaderElectionContextBase {
int leaderVoteWait = cc.getZkController().getLeaderVoteWait();
log.debug("Running the leader process for shard={} and weAreReplacement={} and leaderVoteWait={}", shardId, weAreReplacement, leaderVoteWait);
- // clear the leader in clusterstate
- ZkNodeProps m = new ZkNodeProps(Overseer.QUEUE_OPERATION, OverseerAction.LEADER.toLower(),
- ZkStateReader.SHARD_ID_PROP, shardId, ZkStateReader.COLLECTION_PROP, collection);
- Overseer.getStateUpdateQueue(zkClient).offer(Utils.toJSON(m));
+ if (zkController.getClusterState().getCollection(collection).getSlice(shardId).getReplicas().size() > 1) {
+ // Clear the leader in clusterstate. We only need to worry about this if there is actually more than one replica.
+ ZkNodeProps m = new ZkNodeProps(Overseer.QUEUE_OPERATION, OverseerAction.LEADER.toLower(),
+ ZkStateReader.SHARD_ID_PROP, shardId, ZkStateReader.COLLECTION_PROP, collection);
+ Overseer.getStateUpdateQueue(zkClient).offer(Utils.toJSON(m));
+ }
boolean allReplicasInLine = false;
if (!weAreReplacement) {
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/fd2b4f3f/solr/core/src/test/org/apache/solr/cloud/TestShardsWithSingleReplica.java
----------------------------------------------------------------------
diff --git a/solr/core/src/test/org/apache/solr/cloud/TestShardsWithSingleReplica.java b/solr/core/src/test/org/apache/solr/cloud/TestShardsWithSingleReplica.java
new file mode 100644
index 0000000..a4c51ba
--- /dev/null
+++ b/solr/core/src/test/org/apache/solr/cloud/TestShardsWithSingleReplica.java
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.cloud;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.embedded.JettySolrRunner;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.response.CollectionAdminResponse;
+import org.junit.BeforeClass;
+
+public class TestShardsWithSingleReplica extends SolrCloudTestCase {
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+ System.setProperty("solr.directoryFactory", "solr.StandardDirectoryFactory");
+ System.setProperty("solr.ulog.numRecordsToKeep", "1000");
+
+ configureCluster(3)
+ .addConfig("config", TEST_PATH().resolve("configsets").resolve("cloud-minimal").resolve("conf"))
+ .configure();
+ }
+
+ public void testSkipLeaderOperations() throws Exception {
+ String overseerLeader = getOverseerLeader();
+ List<JettySolrRunner> notOverseerNodes = cluster.getJettySolrRunners()
+ .stream()
+ .filter(solrRunner -> !solrRunner.getNodeName().equals(overseerLeader))
+ .collect(Collectors.toList());
+ String collection = "collection1";
+ CollectionAdminRequest
+ .createCollection(collection, 2, 1)
+ .setCreateNodeSet(notOverseerNodes
+ .stream()
+ .map(JettySolrRunner::getNodeName)
+ .collect(Collectors.joining(","))
+ )
+ .process(cluster.getSolrClient());
+
+ for (JettySolrRunner solrRunner : notOverseerNodes) {
+ cluster.stopJettySolrRunner(solrRunner);
+ }
+ waitForState("Expected empty liveNodes", collection,
+ (liveNodes, collectionState) -> liveNodes.size() == 1);
+
+ CollectionAdminResponse resp = CollectionAdminRequest.getOverseerStatus().process(cluster.getSolrClient());
+ for (JettySolrRunner solrRunner : notOverseerNodes) {
+ cluster.startJettySolrRunner(solrRunner);
+ }
+
+ waitForState("Expected 2x1 for collection: " + collection, collection,
+ clusterShape(2, 1));
+ CollectionAdminResponse resp2 = CollectionAdminRequest.getOverseerStatus().process(cluster.getSolrClient());
+ assertEquals(getNumLeaderOpeations(resp), getNumLeaderOpeations(resp2));
+ }
+
+ private int getNumLeaderOpeations(CollectionAdminResponse resp) {
+ return (int) resp.getResponse().findRecursive("overseer_operations", "leader", "requests");
+ }
+
+ private String getOverseerLeader() throws IOException, SolrServerException {
+ CollectionAdminResponse resp = CollectionAdminRequest.getOverseerStatus().process(cluster.getSolrClient());
+ return (String) resp.getResponse().get("leader");
+ }
+
+}
[41/50] [abbrv] lucene-solr:feature/autoscaling: LUCENE-6673: Maven
build fails for target javadoc:jar.
Posted by no...@apache.org.
LUCENE-6673: Maven build fails for target javadoc:jar.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/0062e999
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/0062e999
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/0062e999
Branch: refs/heads/feature/autoscaling
Commit: 0062e999203fb6dbb7bd9d37e1c4a95cc2367504
Parents: d52564c
Author: Steve Rowe <sa...@apache.org>
Authored: Sun Oct 1 19:03:55 2017 -0400
Committer: Steve Rowe <sa...@apache.org>
Committed: Sun Oct 1 19:03:55 2017 -0400
----------------------------------------------------------------------
dev-tools/maven/pom.xml.template | 5 +++++
lucene/CHANGES.txt | 3 +++
2 files changed, 8 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/0062e999/dev-tools/maven/pom.xml.template
----------------------------------------------------------------------
diff --git a/dev-tools/maven/pom.xml.template b/dev-tools/maven/pom.xml.template
index 6b7f915..c676f6e 100644
--- a/dev-tools/maven/pom.xml.template
+++ b/dev-tools/maven/pom.xml.template
@@ -230,6 +230,11 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9.1</version>
+ <configuration>
+ <quiet>true</quiet>
+ <additionalparam>-Xdoclint:all</additionalparam>
+ <additionalparam>-Xdoclint:-missing</additionalparam>
+ </configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/0062e999/lucene/CHANGES.txt
----------------------------------------------------------------------
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index e0c8124..9cc0fbd 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -86,6 +86,9 @@ Build
* SOLR-11181: Switch order of maven artifact publishing procedure: deploy first
instead of locally installing first, to workaround a double repository push of
*-sources.jar and *-javadoc.jar files. (Lynn Monson via Steve Rowe)
+
+* LUCENE-6673: Maven build fails for target javadoc:jar.
+ (Ramkumar Aiyengar, Daniel Collins via Steve Rowe)
Other
[14/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11165: Merge
docs from issue branch with master for 7.1 editing
Posted by no...@apache.org.
SOLR-11165: Merge docs from issue branch with master for 7.1 editing
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/ee687c39
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/ee687c39
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/ee687c39
Branch: refs/heads/feature/autoscaling
Commit: ee687c39a83bfb5e9fe76b3b145a1719c241a6fe
Parents: 424fb3b
Author: Cassandra Targett <ct...@apache.org>
Authored: Tue Sep 26 14:10:02 2017 -0500
Committer: Cassandra Targett <ct...@apache.org>
Committed: Tue Sep 26 14:10:02 2017 -0500
----------------------------------------------------------------------
.../src/solrcloud-autoscaling-actions.adoc | 28 +++
.../src/solrcloud-autoscaling-api.adoc | 4 +-
...solrcloud-autoscaling-auto-add-replicas.adoc | 21 +++
.../solrcloud-autoscaling-fault-tolerance.adoc | 60 +++++++
.../src/solrcloud-autoscaling-listeners.adoc | 179 +++++++++++++++++++
.../src/solrcloud-autoscaling-overview.adoc | 45 ++++-
.../src/solrcloud-autoscaling-triggers.adoc | 106 +++++++++++
.../src/solrcloud-autoscaling.adoc | 7 +-
8 files changed, 444 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ee687c39/solr/solr-ref-guide/src/solrcloud-autoscaling-actions.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling-actions.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling-actions.adoc
new file mode 100644
index 0000000..676527f
--- /dev/null
+++ b/solr/solr-ref-guide/src/solrcloud-autoscaling-actions.adoc
@@ -0,0 +1,28 @@
+= SolrCloud Autoscaling Actions
+:page-shortname: solrcloud-autoscaling-actions
+:page-permalink: solrcloud-autoscaling-actions.html
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+`TriggerAction` implementations process events generated by triggers in order to ensure the cluster's
+health and good use of resources.
+
+Currently two implementations are provided: `ComputePlanAction` and `ExecutePlanAction`.
+
+== Compute plan action
+
+== Execute plan action
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ee687c39/solr/solr-ref-guide/src/solrcloud-autoscaling-api.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling-api.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling-api.adoc
index 52b5598..2bb3d2a 100644
--- a/solr/solr-ref-guide/src/solrcloud-autoscaling-api.adoc
+++ b/solr/solr-ref-guide/src/solrcloud-autoscaling-api.adoc
@@ -20,11 +20,11 @@
// specific language governing permissions and limitations
// under the License.
-The Autoscaling API is used to manage autoscaling policies and preferences, and to get diagnostics on the state of the cluster.
+The Autoscaling API is used to manage autoscaling policies, preferences, triggers, listeners and to get diagnostics on the state of the cluster.
== Read API
-The autoscaling Read API is available at `/admin/autoscaling` or `/v2/cluster/autoscaling`. It returns information about the configured cluster preferences, cluster policy and collection-specific policies.
+The autoscaling Read API is available at `/admin/autoscaling` or `/v2/cluster/autoscaling`. It returns information about the configured cluster preferences, cluster policy, collection-specific policies triggers and listeners.
This API does not take any parameters.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ee687c39/solr/solr-ref-guide/src/solrcloud-autoscaling-auto-add-replicas.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling-auto-add-replicas.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling-auto-add-replicas.adoc
new file mode 100644
index 0000000..d2a027e
--- /dev/null
+++ b/solr/solr-ref-guide/src/solrcloud-autoscaling-auto-add-replicas.adoc
@@ -0,0 +1,21 @@
+= SolrCloud AutoScaling Automatically Adding Replicas
+:page-shortname: solrcloud-autoscaling-auto-add-replicas
+:page-permalink: solrcloud-autoscaling-auto-add-replicas.html
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+TODO
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ee687c39/solr/solr-ref-guide/src/solrcloud-autoscaling-fault-tolerance.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling-fault-tolerance.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling-fault-tolerance.adoc
new file mode 100644
index 0000000..5792e66
--- /dev/null
+++ b/solr/solr-ref-guide/src/solrcloud-autoscaling-fault-tolerance.adoc
@@ -0,0 +1,60 @@
+= SolrCloud Autoscaling Fault Tolerance
+:page-shortname: solrcloud-autoscaling-fault-tolerance
+:page-permalink: solrcloud-autoscaling-fault-tolerance.html
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+== Node added / lost markers
+Since triggers execute on the node that runs Overseer, should this node go down the `nodeLost`
+event would be lost because there would be no mechanism to generate it. Similarly, if a node has
+been added between the Overseer leader change was completed the `nodeAdded` event would not be
+generated.
+
+For this reason Solr implements additional mechanisms to ensure that these events are generated
+reliably.
+
+When a node joins a cluster its presence is marked as an ephemeral ZK path in the `/live_nodes/<nodeName>`
+ZooKeeper directory, but now also an ephemeral path is created under `/autoscaling/nodeAdded/<nodeName>`.
+When a new instance of Overseer leader is started it will run the `nodeAdded` trigger (if it's configured)
+and discover the presence of this ZK path, at which point it will remove it and generate a `nodeAdded` event.
+
+When a node leaves the cluster up to three remaining nodes will try to create a persistent ZK path
+`/autoscaling/nodeLost/<nodeName>` and eventually one of them succeeds. When a new instance of Overseer leader
+is started it will run the `nodeLost` trigger (if it's configured) and discover the presence of this ZK
+path, at which point it will remove it and generate a `nodeLost` event.
+
+== Trigger state checkpointing
+Triggers generate events based on their internal state. If Overseer leader goes down while the trigger is
+about to generate a new event, it's likely that the event would be lost because a new trigger instance
+running on the new Overseer leader would start from a clean slate.
+
+For this reason after each time a trigger is executed its internal state is persisted to ZooKeeper, and
+on Overseer start its internal state is restored.
+
+== Trigger event queues
+Autoscaling framework limits the rate at which events are processed using several different mechanisms.
+One is the locking mechanism that prevents concurrent
+processing of events, and another is a single-threaded executor that runs trigger actions.
+
+This means that the processing of an event may take significant time, and during this time it's possible that
+Overseer may go down. In order to avoid losing events that were already generated but not yet fully
+processed events are queued before processing is started.
+
+Separate ZooKeeper queues are created for each trigger, and events produced by triggers are put on these
+per-trigger queues. When a new Overseer leader is started it will first check
+these queues and process events accumulated there, and only then it will continue to run triggers
+normally. Queued events that fail processing during this "replay" stage are discarded.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ee687c39/solr/solr-ref-guide/src/solrcloud-autoscaling-listeners.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling-listeners.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling-listeners.adoc
new file mode 100644
index 0000000..e4a3f55
--- /dev/null
+++ b/solr/solr-ref-guide/src/solrcloud-autoscaling-listeners.adoc
@@ -0,0 +1,179 @@
+= SolrCloud AutoScaling Listeners
+:page-shortname: solrcloud-autoscaling-listeners
+:page-permalink: solrcloud-autoscaling-listeners.html
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+Trigger listener API allows users to provide additional behavior related to trigger events as they are being processed.
+
+For example, users may want to record autoscaling events to an external system, or notify administrator when a
+particular type of event occurs, or when its processing reaches certain stage (eg. failed).
+
+Listener configuration always refers to a specific trigger configuration - listener is notified of
+events generated by that specific trigger. Several (or none) named listeners can be registered for a trigger,
+and they will be notified in the order in which they were defined.
+
+Listener configuration can specify what processing stages are of interest - when an event enters this processing stage
+the listener will be notified. Currently the following stages are recognized:
+
+* STARTED - when event has been generated by a trigger and its processing is starting.
+* ABORTED - when event was being processed while the source trigger closed.
+* BEFORE_ACTION - when a `TriggerAction` is about to be invoked. Action name and the current `ActionContext` are passed to the listener.
+* AFTER_ACTION - after a `TriggerAction` has been successfully invoked. Action name, `ActionContext` and the list of action
+ names invoked so far are passed to the listener.
+* FAILED - when event processing failed (or when a `TriggerAction` failed)
+* SUCCEEDED - when event processing completes successfully
+
+Listener configuration can also specify what particular actions are of interest, both
+before and/or after they are invoked.
+
+== Listener configuration
+Currently the following listener configuration properties are supported:
+
+* `name` - (string, required) unique listener configuration name.
+* `trigger` - (string, required) name of an existing trigger configuration.
+* `class` - (string, required) listener implementation class name.
+* `stage` - (list of strings, optional, ignored case) list of processing stages that
+ this listener should be notified. Default is empty list.
+* `beforeAction` - (list of strings, optional) list of action names (as defined in trigger configuration) before
+which the listener will be notified. Default is empty list.
+* `afterAction` - (list of strings, optional) list of action names after which the listener will be notified.
+Default is empty list.
+* additional implementation-specific properties may be provided.
+
+Note: when both `stage` and `beforeAction` / `afterAction` lists are non-empty then the listener will be notified both
+when a specified stage is entered and before / after specified actions.
+
+=== Managing listener configurations
+Listener configurations can be managed using autoscaling Write API, and using `set-listener` and `remove-listener`
+commands.
+
+For example:
+
+[source,json]
+----
+{
+ 'set-listener': {
+ 'name': 'foo',
+ 'trigger': 'node_lost_trigger',
+ 'stage': ['STARTED', 'ABORTED', 'SUCCEEDED', 'FAILED'],
+ 'class': 'solr.SystemLogListener'
+ }
+}
+----
+
+[source,json]
+----
+{
+ 'remove-listener': {
+ 'name': 'foo'
+ }
+}
+----
+
+
+== Listener implementations
+Trigger listeners must implement `TriggerListener` interface. Solr provides some
+implementations of trigger listeners, which cover common use cases. These implementations are described in sections
+below, together with their configuration parameters.
+
+=== `SystemLogListener`
+This trigger listener sends trigger events and processing context as documents for indexing in
+SolrCloud `.system` collection.
+
+Supported configuration properties:
+* `collection` - (string, optional) specifies the target collection where documents are sent.
+Default value is `.system`
+* `enabled` - (boolean, optional) enables the listener when true. Default value is true.
+
+Documents created by this listener have several predefined fields:
+
+* `id` - time-based random id
+* `type` - always set to `autoscaling_event`
+* `source_s` - always set to `SystemLogListener`
+* `timestamp` - current time when document was created
+* `stage_s` - current stage of event processing
+* `action_s` - current action name, if available
+* `message_t` - optional additional message
+* `error.message_t` - message from Throwable, if available
+* `error.details_t` - stacktrace from Throwable, if available
+* `before.actions_ss` - list of action names to be invoked so far
+* `after.actions_ss` - list of action names that have been successfully invoked so far
+* `event_str` - JSON representation of all event properties
+* `context_str` - JSON representation of all `ActionContext` properties, if available
+
+
+The following fields are created using the information from trigger event:
+
+* `event.id_s` - event id
+* `event.type_s` - event type
+* `event.source_s` - event source (trigger name)
+* `event.time_l` - Unix time when the event was created (may significantly differ from the time when it was actually
+processed)
+* `event.property.*` - additional fields that represent other arbitrary event properties. These fields use either
+`_s` or `_ss` suffix depending on whether the property value is a collection (values inside collection are treated as
+strings, there's no recursive flattening)
+
+=== `HttpTriggerListener`
+This listener uses HTTP POST to send a representation of event and context to a specified URL.
+URL, payload and headers may contain property substitution patterns, which are then replaced with values takes from the
+current event or context properties.
+
+Templates use the same syntax as property substitution in Solr configuration files, eg.
+`${foo.bar:baz}` means that the value of `foo.bar` property should be taken, and `baz` should be used
+if the value is absent.
+
+Supported configuration properties:
+
+* `url` - (string, required) a URL template
+* `payload` - (string, optional) payload template. If absent a JSON map of all properties listed above will be used.
+* `contentType` - (string, optional) payload content type. If absent then application/json will be used.
+* `header.*` - (string, optional) header template(s). The name of the property without "header." prefix defines the literal header name.
+* `timeout` - (int, optional) connection and socket timeout in milliseconds. Default is 60 seconds.
+* `followRedirects` - (boolean, optional) setting to follow redirects. Default is false.
+
+The following properties are available in context and can be referenced from templates:
+
+* `config.*` - listener configuration properties
+* `event.*` - current event properties
+* `stage` - current stage of event processing
+* `actionName` - optional current action name
+* `context.*` - optional ActionContext properties
+* `error` - optional error string (from Throwable.toString())
+* `message` - optional message
+
+Example configuration:
+[source,json]
+----
+{
+ 'name': 'foo',
+ 'trigger': 'node_added_trigger',
+ 'class': 'solr.HttpTriggerListener',
+ 'url': 'http://foo.com/${config.name:invalidName}/${config.properties.xyz:invalidXyz}/${event.eventType}',
+ 'xyz': 'foobar',
+ 'header.X-Trigger': '${config.trigger}',
+ 'payload': 'actionName=${actionName}, source=${event.source}, type=${event.eventType}',
+ 'contentType': 'text/plain',
+ 'stage': ['STARTED', 'ABORTED', SUCCEEDED', 'FAILED'],
+ 'beforeAction': ['compute_plan', 'execute_plan'],
+ 'afterAction': ['compute_plan', 'execute_plan']
+}
+----
+This configuration specifies that each time one of the listed stages is reached, or before and after each of the listed
+actions is executed, the listener will send the templated payload to a URL that also depends on the config and the current event,
+and with a custom header that indicates the trigger name.
+
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ee687c39/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc
index d1c9822..dcfe737 100644
--- a/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc
+++ b/solr/solr-ref-guide/src/solrcloud-autoscaling-overview.adoc
@@ -20,7 +20,34 @@
// specific language governing permissions and limitations
// under the License.
-Autoscaling in Solr aims to provide good defaults so a SolrCloud cluster remains balanced and stable in the face of various cluster change events. This balance is achieved by satisfying a set of rules and sorting preferences to select the target of cluster management operations.
+Autoscaling in Solr aims to provide good defaults so a SolrCloud cluster remains balanced and stable in the face of various cluster change events. This balance is achieved by satisfying a set of rules and sorting preferences to select the target of cluster management operations automatically on cluster events.
+
+A simple example is automatically adding a replica for a SolrCloud collection when a node containing an existing replica goes down.
+
+The goal of autoscaling feature is to make SolrCloud cluster management easier, automatic and intelligent. It aims to provide good defaults such that the cluster remains balanced and stable in the face of various events such as a node joining the cluster or leaving the cluster. This is achieved by satisfying a set of rules and sorting preferences that help Solr select the target of cluster management operations.
+
+There are three distinct problems that this feature solves:
+
+* When to run cluster management tasks? e.g. we might want to add a replica when an existing replica is no longer alive.
+* Which cluster management task to run? e.g. do we add a new replica or should we move an existing one to a new node
+* How to run the cluster management tasks such that the cluster remains balanced and stable?
+
+Before we get into the details of how each of these problems are solved, let's take a quick look at the easiest way to setup autoscaling for your cluster.
+
+== QuickStart: Automatically adding replicas
+
+Say that we want to create a collection which always requires us to have three replicas available for each shard all the time. We can set the replicationFactor=3 while creating the collection but what happens if a node containing one or more of the replicas either crashed or was shutdown for maintenance. In such a case, we'd like to create additional replicas to replace the ones that are no longer available to preserve the original number of replicas.
+
+We have an easy way to enable this behavior without needing to understand the autoscaling feature in depth. We can create a collection with such behavior by adding an additional parameter `autoAddReplicas=true` to the create collection API. For example:
+
+`/admin/collections?action=CREATE&name=_name_of_collection_&numShards=1&replicationFactor=3&autoAddReplicas=true`
+
+A collection created with `autoAddReplicas=true` will be monitored by Solr such that if a node containing a replica of this collection goes down, Solr will add new replicas on other nodes after waiting for up to thirty seconds for the node to come back.
+
+You can see the section __TODO_FIX_ME__ to learn more about how to enable or disable this feature as well as other details.
+
+The selection of the node that will host the new replica is made according to the default cluster preferences that we will learn more about in the next sections.
+
== Cluster Preferences
@@ -50,7 +77,21 @@ The above create collection command will associate a policy named `policy1` with
Note that the collection-specific policy is applied *in addition to* the cluster policy, i.e., it is not an override but an augmentation. Therefore the collection will follow all conditions laid out in the cluster preferences, cluster policy, and the policy named `policy1`.
-You can learn more about collection-specific policies in the section <<solrcloud-autoscaling-policy-preferences.adoc#collection-specific-policy,Defining Collection-Specific Policies>>.
+You can learn more about collection-specific policies in the section <<solrcloud-autoscaling-policy-preferences.adoc#defining-collection-specific-policies,Defining Collection-Specific Policies>>.
+
+== Triggers
+
+Now that we have an idea about how cluster management operations use policy and preferences help Solr keep the cluster balanced and stable, we can talk about when to invoke such operations. Triggers are used to watch for events such as a node joining or leaving the cluster. When the event happens, the trigger executes a set of `actions` that compute and execute a *plan* i.e. a set of operations to change the cluster so that the policy and preferences are respected.
+
+The `autoAddReplicas` parameter passed to the create collection API in the quickstart section automatically creates a trigger that watches for a node going away. When the trigger fires, it computes and executes a plan to move all replicas hosted by the lost node to new nodes in the cluster. The target nodes are chosen based on the policy and preferences.
+
+You can learn more about Triggers in the __TODO__ section.
+
+== Listeners
+
+An AutoScaling *Listener* is attached to a trigger. Solr calls the listener each time the trigger fires as well as before and after the actions performed by the trigger. Listeners are useful as a call back mechanism to perform tasks such as logging or informing external systems about events. For example, a listener is automatically added by Solr to each Trigger to log details of the trigger fire and actions to the `.system` collection.
+
+You can learn more about Listeners in the __TODO__ section.
== Autoscaling APIs
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ee687c39/solr/solr-ref-guide/src/solrcloud-autoscaling-triggers.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling-triggers.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling-triggers.adoc
new file mode 100644
index 0000000..38945f1
--- /dev/null
+++ b/solr/solr-ref-guide/src/solrcloud-autoscaling-triggers.adoc
@@ -0,0 +1,106 @@
+= SolrCloud AutoScaling Triggers
+:page-shortname: solrcloud-autoscaling-triggers
+:page-permalink: solrcloud-autoscaling-triggers.html
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+Triggers are used by autoscaling API to watch for cluster events such as node joining or leaving,
+and in the future also for other cluster, node and replica events that are important from the
+point of view of cluster performance.
+
+Trigger implementations verify the state of resources that they monitor. When they detect a
+change that merits attention they generate events, which are then queued and processed by configured
+`TriggerAction` implementations - this usually involves computing and executing a plan to manage the new cluster
+resources (eg. move replicas). Solr provides predefined implementations of triggers for specific event types.
+
+Triggers execute on the node that runs `Overseer`. They are scheduled to run periodically,
+currently at fixed interval of 1s between each execution (not every execution produces events).
+
+== Event types
+Currently the following event types (and corresponding trigger implementations) are defined:
+
+* `nodeAdded` - generated when a new node joins the cluster
+* `nodeLost` - generated when a node leaves the cluster
+
+Events are not necessarily generated immediately after the corresponding state change occurred - the
+maximum rate of events is controlled by the `waitFor` configuration parameter (see below).
+
+The following properties are common to all event types:
+
+* `id` - (string) unique time-based event id.
+* `eventType` - (string) event type.
+* `source` - (string) name of the trigger that produced this event.
+* `eventTime` - (long) Unix time when the condition that caused this event occurred. For example, for
+`nodeAdded` event this will be the time when the node was added and not when the event was actually
+generated, which may significantly differ due to the rate limits set by `waitFor`.
+* `properties` - (map, optional) additional properties. Currently contains `nodeName` property that
+indicates the node that was lost or added.
+
+== `.autoAddReplicas` trigger
+When a collection has a flag `autoAddReplicas` set to true then a trigger configuration named `.auto_add_replicas`
+is automatically created to watch for nodes going away. This trigger produces `nodeLost` events,
+which are then processed by configured actions (usually resulting in computing and executing a plan
+to add replicas on the live nodes to maintain the expected replication factor).
+
+== Trigger configuration
+Trigger configurations are managed using autoscaling Write API with commands `set-trigger`, `remove-trigger`,
+`suspend-trigger`, `resume-trigger`.
+
+Trigger configuration consists of the following properties:
+
+* `name` - (string, required) unique trigger configuration name.
+* `event` - (string, required) one of predefined event types (nodeAdded, nodeLost).
+* `actions` - (list of action configs, optional) ordered list of actions to execute when event is fired
+* `waitFor` - (string, optional) time to wait between generating new events, as an integer number immediately followed
+by unit symbol, one of "s" (seconds), "m" (minutes), or "h" (hours). Default is "0s".
+* `enabled` - (boolean, optional) when true the trigger is enabled. Default is true.
+* additional implementation-specific properties may be provided
+
+Action configuration consists of the following properties:
+
+* `name` - (string, required) unique name of the action configuration.
+* `class` - (string, required) action implementation class
+* additional implementation-specific properties may be provided
+
+Example: adding / updating a trigger for `nodeAdded` events. This trigger configuration will
+compute and execute a plan to allocate the resources available on the new node. A custom action
+is also used to possibly modify the plan.
+[source,json]
+----
+{
+ 'set-trigger': {
+ 'name' : 'node_lost_trigger',
+ 'event' : 'nodeLost',
+ 'waitFor' : '1s',
+ 'enabled' : true,
+ 'actions' : [
+ {
+ 'name' : 'compute_plan',
+ 'class': 'solr.ComputePlanAction'
+ },
+ {
+ 'name' : 'custom_action',
+ 'class': 'com.example.CustomAction'
+ },
+ {
+ 'name' : 'execute_plan',
+ 'class': 'solr.ExecutePlanAction'
+ }
+ ]
+ }
+}
+----
+
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ee687c39/solr/solr-ref-guide/src/solrcloud-autoscaling.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/solrcloud-autoscaling.adoc b/solr/solr-ref-guide/src/solrcloud-autoscaling.adoc
index 457120b..3c0d2c5 100644
--- a/solr/solr-ref-guide/src/solrcloud-autoscaling.adoc
+++ b/solr/solr-ref-guide/src/solrcloud-autoscaling.adoc
@@ -1,7 +1,7 @@
-= SolrCloud Autoscaling
+= SolrCloud AutoScaling
:page-shortname: solrcloud-autoscaling
:page-permalink: solrcloud-autoscaling.html
-:page-children: solrcloud-autoscaling-overview, solrcloud-autoscaling-policy-preferences, solrcloud-autoscaling-api
+:page-children: solrcloud-autoscaling-overview, solrcloud-autoscaling-api, solrcloud-autoscaling-policy-preferences, solrcloud-autoscaling-triggers, solrcloud-autoscaling-listeners, solrcloud-autoscaling-auto-add-replicas
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
@@ -29,3 +29,6 @@ The following sections describe the autoscaling features of SolrCloud:
* <<solrcloud-autoscaling-overview.adoc#solrcloud-autoscaling-overview,Overview of Autoscaling in SolrCloud>>
* <<solrcloud-autoscaling-api.adoc#solrcloud-autoscaling-api,SolrCloud Autoscaling API>>
* <<solrcloud-autoscaling-policy-preferences.adoc#solrcloud-autoscaling-policy-preferences,SolrCloud Autoscaling Policy and Preferences>>
+* <<solrcloud-autoscaling-triggers.adoc#solrcloud-autoscaling-triggers,SolrCloud AutoScaling Triggers>>
+* <<solrcloud-autoscaling-listeners.adoc#solrcloud-autoscaling-listeners,SolrCloud AutoScaling Listeners>>
+* <<solrcloud-autoscaling-auto-add-replicas.adoc#solrcloud-autoscaling-auto-add-replicas,SolrCloud AutoScaling - Automatically Adding Replicas>>
[13/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11224:
SolrStream.close can hit an NPE
Posted by no...@apache.org.
SOLR-11224: SolrStream.close can hit an NPE
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/424fb3bf
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/424fb3bf
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/424fb3bf
Branch: refs/heads/feature/autoscaling
Commit: 424fb3bfb762fff0a46b12c75394e1c1fed011bb
Parents: 3b0f46b
Author: Erick Erickson <er...@apache.org>
Authored: Tue Sep 26 09:26:12 2017 -0700
Committer: Erick Erickson <er...@apache.org>
Committed: Tue Sep 26 09:26:12 2017 -0700
----------------------------------------------------------------------
solr/CHANGES.txt | 2 ++
.../java/org/apache/solr/client/solrj/io/stream/SolrStream.java | 4 +++-
2 files changed, 5 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/424fb3bf/solr/CHANGES.txt
----------------------------------------------------------------------
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txt
index 2e1c70d..7cd3717 100644
--- a/solr/CHANGES.txt
+++ b/solr/CHANGES.txt
@@ -151,6 +151,8 @@ Bug Fixes
* SOLR-11399: The UnifiedHighlighter was ignoring the hl.fragsize parameter when hl.bs.type=SEPARATOR
(Marc Morissette via David Smiley)
+* SOLR-11224: SolrStream.close can hit an NPE (Erick Erickson)
+
Optimizations
----------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/424fb3bf/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/SolrStream.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/SolrStream.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/SolrStream.java
index 17817bd..22569a4 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/SolrStream.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/SolrStream.java
@@ -176,7 +176,9 @@ public class SolrStream extends TupleStream {
* */
public void close() throws IOException {
- closeableHttpResponse.close();
+ if (closeableHttpResponse != null) {
+ closeableHttpResponse.close();
+ }
if(cache == null) {
client.close();
}
[36/50] [abbrv] lucene-solr:feature/autoscaling:
addBackcompatIndexes.py: 4 space indent instead of 6 for new index names;
check previous line when deciding whether there are previous versions.
Posted by no...@apache.org.
addBackcompatIndexes.py: 4 space indent instead of 6 for new index names; check previous line when deciding whether there are previous versions.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/0e8d9411
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/0e8d9411
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/0e8d9411
Branch: refs/heads/feature/autoscaling
Commit: 0e8d9411b0714e581793689530c5e43ef5939950
Parents: 5afd41d
Author: Steve Rowe <sa...@gmail.com>
Authored: Sat Sep 30 23:20:51 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Sat Sep 30 23:20:51 2017 -0400
----------------------------------------------------------------------
dev-tools/scripts/addBackcompatIndexes.py | 6 +++---
.../org/apache/lucene/index/TestBackwardsCompatibility.java | 6 +++---
2 files changed, 6 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/0e8d9411/dev-tools/scripts/addBackcompatIndexes.py
----------------------------------------------------------------------
diff --git a/dev-tools/scripts/addBackcompatIndexes.py b/dev-tools/scripts/addBackcompatIndexes.py
index 1cada01..03e858c 100644
--- a/dev-tools/scripts/addBackcompatIndexes.py
+++ b/dev-tools/scripts/addBackcompatIndexes.py
@@ -121,8 +121,8 @@ def update_backcompat_tests(types, index_version, current_version):
def __call__(self, buffer, match, line):
if self.start:
# find where this version should exist
- i = len(buffer) - 1
- previous_version_exists = '};' not in line # Version list closure => there are no versions
+ i = len(buffer) - 1
+ previous_version_exists = not ('};' in line and buffer[-1].strip().endswith("{"))
if previous_version_exists: # Only look if there is a version here
v = find_version(buffer[i])
while i >= self.start and v.on_or_after(index_version):
@@ -143,7 +143,7 @@ def update_backcompat_tests(types, index_version, current_version):
last = buffer[-1]
spaces = ' ' * (len(last) - len(last.lstrip()))
else:
- spaces = ' '
+ spaces = ' '
for (j, t) in enumerate(types):
if t == 'sorted':
newline = spaces + ('"sorted.%s"') % index_version
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/0e8d9411/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
----------------------------------------------------------------------
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
index 88900d4..7437e75 100644
--- a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
+++ b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
@@ -287,8 +287,8 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
}
final static String[] oldNames = {
- "7.0.0-cfs",
- "7.0.0-nocfs"
+ "7.0.0-cfs",
+ "7.0.0-nocfs"
};
public static String[] getOldNames() {
@@ -296,7 +296,7 @@ public class TestBackwardsCompatibility extends LuceneTestCase {
}
final static String[] oldSortedNames = {
- "sorted.7.0.0"
+ "sorted.7.0.0"
};
public static String[] getOldSortedNames() {
[16/50] [abbrv] lucene-solr:feature/autoscaling: Merge branch
'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr
Posted by no...@apache.org.
Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/lucene-solr
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/c05d34d6
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/c05d34d6
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/c05d34d6
Branch: refs/heads/feature/autoscaling
Commit: c05d34d6f9618e8b79fa23d7abb1f4789e04104c
Parents: cc16055 ee687c39
Author: Karl Wright <Da...@gmail.com>
Authored: Wed Sep 27 04:31:34 2017 -0400
Committer: Karl Wright <Da...@gmail.com>
Committed: Wed Sep 27 04:31:34 2017 -0400
----------------------------------------------------------------------
solr/CHANGES.txt | 5 +
.../solr/highlight/UnifiedSolrHighlighter.java | 18 +-
.../highlight/TestUnifiedSolrHighlighter.java | 3 +
.../src/rule-based-authorization-plugin.adoc | 6 +-
.../src/solrcloud-autoscaling-actions.adoc | 28 +++
.../src/solrcloud-autoscaling-api.adoc | 4 +-
...solrcloud-autoscaling-auto-add-replicas.adoc | 21 +++
.../solrcloud-autoscaling-fault-tolerance.adoc | 60 +++++++
.../src/solrcloud-autoscaling-listeners.adoc | 179 +++++++++++++++++++
.../src/solrcloud-autoscaling-overview.adoc | 45 ++++-
.../src/solrcloud-autoscaling-triggers.adoc | 106 +++++++++++
.../src/solrcloud-autoscaling.adoc | 7 +-
.../solr/client/solrj/io/stream/SolrStream.java | 4 +-
13 files changed, 469 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
[07/50] [abbrv] lucene-solr:feature/autoscaling: Ref Guide: rename
Streaming docs to fix broken intra-doc links in the PDF due to mismatching
anchor names
Posted by no...@apache.org.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9beafb61/solr/solr-ref-guide/src/stream-decorators.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/stream-decorators.adoc b/solr/solr-ref-guide/src/stream-decorators.adoc
deleted file mode 100644
index 2afb524..0000000
--- a/solr/solr-ref-guide/src/stream-decorators.adoc
+++ /dev/null
@@ -1,1216 +0,0 @@
-= Stream Decorator Reference
-:page-shortname: stream-decorators
-:page-permalink: stream-decorators.html
-:page-tocclass: right
-:page-toclevels: 1
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-== cartesianProduct
-
-The `cartesianProduct` function turns a single tuple with a multi-valued field (ie. an array) into multiple tuples, one for each value in the array field. That is, given a single tuple containing an array of N values for fieldA, the `cartesianProduct` function will output N tuples, each with one value from the original tuple's array. In essence, you can flatten arrays for further processing.
-
-For example, using `cartesianProduct` you can turn this tuple
-[source,text]
-----
-{
- "fieldA": "foo",
- "fieldB": ["bar","baz","bat"]
-}
-----
-
-into the following 3 tuples
-[source,text]
-----
-{
- "fieldA": "foo",
- "fieldB": "bar"
-}
-{
- "fieldA": "foo",
- "fieldB": "baz"
-}
-{
- "fieldA": "foo",
- "fieldB": "bat"
-}
-----
-
-=== cartesianProduct Parameters
-
-* `incoming stream`: (Mandatory) A single incoming stream.
-* `fieldName or evaluator`: (Mandatory) Name of field to flatten values for, or evaluator whose result should be flattened.
-* `productSort='fieldName ASC|DESC'`: (Optional) Sort order of the newly generated tuples.
-
-=== cartesianProduct Syntax
-
-[source,text]
-----
-cartesianProduct(
- <stream>,
- <fieldName | evaluator> [as newFieldName],
- productSort='fieldName ASC|DESC'
-)
-----
-
-=== cartesianProduct Examples
-
-The following examples show different outputs for this source tuple
-
-[source,text]
-----
-{
- "fieldA": "valueA",
- "fieldB": ["valueB1","valueB2"],
- "fieldC": [1,2,3]
-}
-----
-
-==== Single Field, No Sorting
-
-[source,text]
-----
-cartesianProduct(
- search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
- fieldB
-)
-
-{
- "fieldA": "valueA",
- "fieldB": "valueB1",
- "fieldC": [1,2,3]
-}
-{
- "fieldA": "valueA",
- "fieldB": "valueB2",
- "fieldC": [1,2,3]
-}
-----
-
-==== Single Evaluator, No Sorting
-
-[source,text]
-----
-cartesianProduct(
- search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
- sequence(3,4,5) as fieldE
-)
-
-{
- "fieldA": "valueA",
- "fieldB": ["valueB1","valueB2"],
- "fieldC": [1,2,3],
- "fieldE": 4
-}
-{
- "fieldA": "valueA",
- "fieldB": ["valueB1","valueB2"],
- "fieldC": [1,2,3],
- "fieldE": 9
-}
-{
- "fieldA": "valueA",
- "fieldB": ["valueB1","valueB2"],
- "fieldC": [1,2,3],
- "fieldE": 14
-}
-----
-
-==== Single Field, Sorted by Value
-
-[source,text]
-----
-cartesianProduct(
- search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
- fieldB,
- productSort="fieldB DESC"
-)
-
-{
- "fieldA": "valueA",
- "fieldB": "valueB2",
- "fieldC": [1,2,3]
-}
-{
- "fieldA": "valueA",
- "fieldB": "valueB1",
- "fieldC": [1,2,3]
-}
-----
-
-==== Single Evaluator, Sorted by Evaluator Values
-
-[source,text]
-----
-cartesianProduct(
- search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
- sequence(3,4,5) as fieldE,
- productSort='newFieldE DESC'
-)
-
-{
- "fieldA": "valueA",
- "fieldB": ["valueB1","valueB2"],
- "fieldC": [1,2,3],
- "fieldE": 14
-}
-{
- "fieldA": "valueA",
- "fieldB": ["valueB1","valueB2"],
- "fieldC": [1,2,3],
- "fieldE": 9
-}
-{
- "fieldA": "valueA",
- "fieldB": ["valueB1","valueB2"],
- "fieldC": [1,2,3],
- "fieldE": 4
-}
-----
-
-==== Renamed Single Field, Sorted by Value
-
-[source,text]
-----
-cartesianProduct(
- search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
- fieldB as newFieldB,
- productSort="fieldB DESC"
-)
-
-{
- "fieldA": "valueA",
- "fieldB": ["valueB1","valueB2"],
- "fieldC": [1,2,3]
- "newFieldB": "valueB2",
-}
-{
- "fieldA": "valueA",
- "fieldB": ["valueB1","valueB2"],
- "fieldC": [1,2,3]
- "newFieldB": "valueB1",
-}
-----
-
-==== Multiple Fields, No Sorting
-
-[source,text]
-----
-cartesianProduct(
- search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
- fieldB,
- fieldC
-)
-
-{
- "fieldA": "valueA",
- "fieldB": "valueB1",
- "fieldC": 1
-}
-{
- "fieldA": "valueA",
- "fieldB": "valueB1",
- "fieldC": 2
-}
-{
- "fieldA": "valueA",
- "fieldB": "valueB1",
- "fieldC": 3
-}
-{
- "fieldA": "valueA",
- "fieldB": "valueB2",
- "fieldC": 1
-}
-{
- "fieldA": "valueA",
- "fieldB": "valueB2",
- "fieldC": 2
-}
-{
- "fieldA": "valueA",
- "fieldB": "valueB2",
- "fieldC": 3
-}
-----
-
-==== Multiple Fields, Sorted by Single Field
-
-[source,text]
-----
-cartesianProduct(
- search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
- fieldB,
- fieldC,
- productSort="fieldC ASC"
-)
-
-{
- "fieldA": "valueA",
- "fieldB": "valueB1",
- "fieldC": 1
-}
-{
- "fieldA": "valueA",
- "fieldB": "valueB2",
- "fieldC": 1
-}
-{
- "fieldA": "valueA",
- "fieldB": "valueB1",
- "fieldC": 2
-}
-{
- "fieldA": "valueA",
- "fieldB": "valueB2",
- "fieldC": 2
-}
-{
- "fieldA": "valueA",
- "fieldB": "valueB1",
- "fieldC": 3
-}
-{
- "fieldA": "valueA",
- "fieldB": "valueB2",
- "fieldC": 3
-}
-----
-
-==== Multiple Fields, Sorted by Multiple Fields
-
-[source,text]
-----
-cartesianProduct(
- search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
- fieldB,
- fieldC,
- productSort="fieldC ASC, fieldB DESC"
-)
-
-{
- "fieldA": "valueA",
- "fieldB": "valueB2",
- "fieldC": 1
-}
-{
- "fieldA": "valueA",
- "fieldB": "valueB1",
- "fieldC": 1
-}
-{
- "fieldA": "valueA",
- "fieldB": "valueB2",
- "fieldC": 2
-}
-{
- "fieldA": "valueA",
- "fieldB": "valueB1",
- "fieldC": 2
-}
-{
- "fieldA": "valueA",
- "fieldB": "valueB2",
- "fieldC": 3
-}
-{
- "fieldA": "valueA",
- "fieldB": "valueB1",
- "fieldC": 3
-}
-----
-
-==== Field and Evaluator, No Sorting
-
-[source,text]
-----
-cartesianProduct(
- search(collection1, q='*:*', fl='fieldA, fieldB, fieldC', sort='fieldA ASC'),
- sequence(3,4,5) as fieldE,
- fieldB
-)
-
-{
- "fieldA": "valueA",
- "fieldB": valueB1,
- "fieldC": [1,2,3],
- "fieldE": 4
-}
-{
- "fieldA": "valueA",
- "fieldB": valueB2,
- "fieldC": [1,2,3],
- "fieldE": 4
-}
-{
- "fieldA": "valueA",
- "fieldB": valueB1,
- "fieldC": [1,2,3],
- "fieldE": 9
-}
-{
- "fieldA": "valueA",
- "fieldB": valueB2,
- "fieldC": [1,2,3],
- "fieldE": 9
-}
-{
- "fieldA": "valueA",
- "fieldB": valueB1,
- "fieldC": [1,2,3],
- "fieldE": 14
-}
-{
- "fieldA": "valueA",
- "fieldB": valueB2,
- "fieldC": [1,2,3],
- "fieldE": 14
-}
-----
-
-As you can see in the examples above, the `cartesianProduct` function does support flattening tuples across multiple fields and/or evaluators.
-
-== classify
-
-The `classify` function classifies tuples using a logistic regression text classification model. It was designed specifically to work with models trained using the <<stream-sources.adoc#train,train function>>. The `classify` function uses the <<stream-sources.adoc#model,model function>> to retrieve a stored model and then scores a stream of tuples using the model. The tuples read by the classifier must contain a text field that can be used for classification. The classify function uses a Lucene analyzer to extract the features from the text so the model can be applied. By default the `classify` function looks for the analyzer using the name of text field in the tuple. If the Solr schema on the worker node does not contain this field, the analyzer can be looked up in another field by specifying the `analyzerField` parameter.
-
-Each tuple that is classified is assigned two scores:
-
-* probability_d* : A float between 0 and 1 which describes the probability that the tuple belongs to the class. This is useful in the classification use case.
-
-* score_d* : The score of the document that has not be squashed between 0 and 1. The score may be positive or negative. The higher the score the better the document fits the class. This un-squashed score will be useful in query re-ranking and recommendation use cases. This score is particularly useful when multiple high ranking documents have a probability_d score of 1, which won't provide a meaningful ranking between documents.
-
-=== classify Parameters
-
-* `model expression`: (Mandatory) Retrieves the stored logistic regression model.
-* `field`: (Mandatory) The field in the tuples to apply the classifier to. By default the analyzer for this field in the schema will be used extract the features.
-* `analyzerField`: (Optional) Specifies a different field to find the analyzer from in the schema.
-
-=== classify Syntax
-
-[source,text]
-----
-classify(model(modelCollection,
- id="model1",
- cacheMillis=5000),
- search(contentCollection,
- q="id:(a b c)",
- fl="text_t, id",
- sort="id asc"),
- field="text_t")
-----
-
-In the example above the `classify expression` is retrieving the model using the `model` function. It is then classifying tuples returned by the `search` function. The `text_t` field is used for the text classification and the analyzer for the `text_t` field in the Solr schema is used to analyze the text and extract the features.
-
-== commit
-
-The `commit` function wraps a single stream (A) and given a collection and batch size will send commit messages to the collection when the batch size is fulfilled or the end of stream is reached. A commit stream is used most frequently with an update stream and as such the commit will take into account possible summary tuples coming from the update stream. All tuples coming into the commit stream will be returned out of the commit stream - no tuples will be dropped and no tuples will be added.
-
-=== commit Parameters
-
-* `collection`: The collection to send commit messages to (required)
-* `batchSize`: The commit batch size, sends commit message when batch size is hit. If not provided (or provided as value 0) then a commit is only sent at the end of the incoming stream.
-* `waitFlush`: The value passed directly to the commit handler (true/false, default: false)
-* `waitSearcher`: The value passed directly to the commit handler (true/false, default: false)
-* `softCommit`: The value passed directly to the commit handler (true/false, default: false)
-* `StreamExpression for StreamA` (required)
-
-=== commit Syntax
-
-[source,text]
-----
-commit(
- destinationCollection,
- batchSize=2,
- update(
- destinationCollection,
- batchSize=5,
- search(collection1, q=*:*, fl="id,a_s,a_i,a_f,s_multi,i_multi", sort="a_f asc, a_i asc")
- )
-)
-----
-
-== complement
-
-The `complement` function wraps two streams (A and B) and emits tuples from A which do not exist in B. The tuples are emitted in the order in which they appear in stream A. Both streams must be sorted by the fields being used to determine equality (using the `on` parameter).
-
-=== complement Parameters
-
-* `StreamExpression for StreamA`
-* `StreamExpression for StreamB`
-* `on`: Fields to be used for checking equality of tuples between A and B. Can be of the format `on="fieldName"`, `on="fieldNameInLeft=fieldNameInRight"`, or `on="fieldName, otherFieldName=rightOtherFieldName"`.
-
-=== complement Syntax
-
-[source,text]
-----
-complement(
- search(collection1, q=a_s:(setA || setAB), fl="id,a_s,a_i", sort="a_i asc, a_s asc"),
- search(collection1, q=a_s:(setB || setAB), fl="id,a_s,a_i", sort="a_i asc"),
- on="a_i"
-)
-
-complement(
- search(collection1, q=a_s:(setA || setAB), fl="id,a_s,a_i", sort="a_i asc, a_s asc"),
- search(collection1, q=a_s:(setB || setAB), fl="id,a_s,a_i", sort="a_i asc, a_s asc"),
- on="a_i,a_s"
-)
-----
-
-== daemon
-
-The `daemon` function wraps another function and runs it at intervals using an internal thread. The `daemon` function can be used to provide both continuous push and pull streaming.
-
-=== Continuous Push Streaming
-
-With continuous push streaming the `daemon` function wraps another function and is then sent to the `/stream` handler for execution. The `/stream` handler recognizes the `daemon` function and keeps it resident in memory, so it can run its internal function at intervals.
-
-In order to facilitate the pushing of tuples, the `daemon` function must wrap another stream decorator that pushes the tuples somewhere. One example of this is the `update` function, which wraps a stream and sends the tuples to another SolrCloud collection for indexing.
-
-=== daemon Syntax
-
-[source,text]
-----
-daemon(id="uniqueId",
- runInterval="1000",
- terminate="true",
- update(destinationCollection,
- batchSize=100,
- topic(checkpointCollection,
- topicCollection,
- q="topic query",
- fl="id, title, abstract, text",
- id="topicId",
- initialCheckpoint=0)
- )
- )
-----
-
-The sample code above shows a `daemon` function wrapping an `update` function, which is wrapping a `topic` function. When this expression is sent to the `/stream` handler, the `/stream` hander sees the `daemon` function and keeps it in memory where it will run at intervals. In this particular example, the `daemon` function will run the `update` function every second. The `update` function is wrapping a <<stream-sources.adoc#topic,`topic` function>>, which will stream tuples that match the `topic` function query in batches. Each subsequent call to the topic will return the next batch of tuples for the topic. The `update` function will send all the tuples matching the topic to another collection to be indexed. The `terminate` parameter tells the daemon to terminate when the `topic` function stops sending tuples.
-
-The effect of this is to push documents that match a specific query into another collection. Custom push functions can be plugged in that push documents out of Solr and into other systems, such as Kafka or an email system.
-
-Push streaming can also be used for continuous background aggregation scenarios where aggregates are rolled up in the background at intervals and pushed to other Solr collections. Another use case is continuous background machine learning model optimization, where the optimized model is pushed to another Solr collection where it can be integrated into queries.
-
-The `/stream` handler supports a small set commands for listing and controlling daemon functions:
-
-[source,text]
-----
-http://localhost:8983/collection/stream?action=list
-----
-
-This command will provide a listing of the current daemon's running on the specific node along with there current state.
-
-[source,text]
-----
-http://localhost:8983/collection/stream?action=stop&id=daemonId
-----
-
-This command will stop a specific daemon function but leave it resident in memory.
-
-[source,text]
-----
-http://localhost:8983/collection/stream?action=start&id=daemonId
-----
-
-This command will start a specific daemon function that has been stopped.
-
-[source,text]
-----
-http://localhost:8983/collection/stream?action=kill&id=daemonId
-----
-
-This command will stop a specific daemon function and remove it from memory.
-
-=== Continuous Pull Streaming
-
-The {solr-javadocs}/solr-solrj/org/apache/solr/client/solrj/io/stream/DaemonStream.html[DaemonStream] java class (part of the SolrJ libraries) can also be embedded in a java application to provide continuous pull streaming. Sample code:
-
-[source,java]
-----
-StreamContext context = new StreamContext()
-SolrClientCache cache = new SolrClientCache();
-context.setSolrClientCache(cache);
-
-Map topicQueryParams = new HashMap();
-topicQueryParams.put("q","hello"); // The query for the topic
-topicQueryparams.put("rows", "500"); // How many rows to fetch during each run
-topicQueryparams.put("fl", "id", "title"); // The field list to return with the documents
-
-TopicStream topicStream = new TopicStream(zkHost, // Host address for the zookeeper service housing the collections
- "checkpoints", // The collection to store the topic checkpoints
- "topicData", // The collection to query for the topic records
- "topicId", // The id of the topic
- -1, // checkpoint every X tuples, if set -1 it will checkpoint after each run.
- topicQueryParams); // The query parameters for the TopicStream
-
-DaemonStream daemonStream = new DaemonStream(topicStream, // The underlying stream to run.
- "daemonId", // The id of the daemon
- 1000, // The interval at which to run the internal stream
- 500); // The internal queue size for the daemon stream. Tuples will be placed in the queue
- // as they are read by the internal internal thread.
- // Calling read() on the daemon stream reads records from the internal queue.
-
-daemonStream.setStreamContext(context);
-
-daemonStream.open();
-
-//Read until it's time to shutdown the DaemonStream. You can define the shutdown criteria.
-while(!shutdown()) {
- Tuple tuple = daemonStream.read() // This will block until tuples become available from the underlying stream (TopicStream)
- // The EOF tuple (signaling the end of the stream) will never occur until the DaemonStream has been shutdown.
- //Do something with the tuples
-}
-
-// Shutdown the DaemonStream.
-daemonStream.shutdown();
-
-//Read the DaemonStream until the EOF Tuple is found.
-//This allows the underlying stream to perform an orderly shutdown.
-
-while(true) {
- Tuple tuple = daemonStream.read();
- if(tuple.EOF) {
- break;
- } else {
- //Do something with the tuples.
- }
-}
-//Finally close the stream
-daemonStream.close();
-----
-
-== eval
-
-The `eval` function allows for use cases where new streaming expressions are generated on the fly and then evaluated.
-The `eval` function wraps a streaming expression and reads a single tuple from the underlying stream.
-The `eval` function then retrieves a string Streaming Expressions from the `expr_s` field of the tuple.
-The `eval` function then compiles the string Streaming Expression and emits the tuples.
-
-=== eval Parameters
-
-* `StreamExpression`: (Mandatory) The stream which provides the streaming expression to be evaluated.
-
-=== eval Syntax
-
-[source,text]
-----
-eval(expr)
-----
-
-In the example above the `eval` expression reads the first tuple from the underlying expression. It then compiles and
-executes the string Streaming Expression in the epxr_s field.
-
-== executor
-
-The `executor` function wraps a stream source that contains streaming expressions, and executes the expressions in parallel. The `executor` function looks for the expression in the `expr_s` field in each tuple. The `executor` function has an internal thread pool that runs tasks that compile and run expressions in parallel on the same worker node. This function can also be parallelized across worker nodes by wrapping it in the <<parallel,`parallel`>> function to provide parallel execution of expressions across a cluster.
-
-The `executor` function does not do anything specific with the output of the expressions that it runs. Therefore the expressions that are executed must contain the logic for pushing tuples to their destination. The <<update,update function>> can be included in the expression being executed to send the tuples to a SolrCloud collection for storage.
-
-This model allows for asynchronous execution of jobs where the output is stored in a SolrCloud collection where it can be accessed as the job progresses.
-
-=== executor Parameters
-
-* `threads`: (Optional) The number of threads in the executors thread pool for executing expressions.
-* `StreamExpression`: (Mandatory) The stream source which contains the Streaming Expressions to execute.
-
-=== executor Syntax
-
-[source,text]
-----
-daemon(id="myDaemon",
- terminate="true",
- executor(threads=10,
- topic(checkpointCollection
- storedExpressions,
- q="*:*",
- fl="id, expr_s",
- initialCheckPoint=0,
- id="myTopic")))
-----
-
-In the example above a <<daemon,daemon>> wraps an executor, which wraps a <<stream-sources.adoc#topic,topic>> that is returning tuples with expressions to execute. When sent to the stream handler, the daemon will call the executor at intervals which will cause the executor to read from the topic and execute the expressions found in the `expr_s` field. The daemon will repeatedly call the executor until all the tuples that match the topic have been iterated, then it will terminate. This is the approach for executing batches of streaming expressions from a `topic` queue.
-
-== fetch
-
-The `fetch` function iterates a stream and fetches additional fields and adds them to the tuples. The `fetch` function fetches in batches to limit the number of calls back to Solr. Tuples streamed from the `fetch` function will contain the original fields and the additional fields that were fetched. The `fetch` function supports one-to-one fetches. Many-to-one fetches, where the stream source contains duplicate keys, will also work, but one-to-many fetches are currently not supported by this function.
-
-=== fetch Parameters
-
-* `Collection`: (Mandatory) The collection to fetch the fields from.
-* `StreamExpression`: (Mandatory) The stream source for the fetch function.
-* `fl`: (Mandatory) The fields to be fetched.
-* `on`: Fields to be used for checking equality of tuples between stream source and fetched records. Formatted as `on="fieldNameInTuple=fieldNameInCollection"`.
-* `batchSize`: (Optional) The batch fetch size.
-
-=== fetch Syntax
-
-[source,text]
-----
-fetch(addresses,
- search(people, q="*:*", fl="username, firstName, lastName", sort="username asc"),
- fl="streetAddress, city, state, country, zip",
- on="username=userId")
-----
-
-The example above fetches addresses for users by matching the username in the tuple with the userId field in the addresses collection.
-
-== having
-
-The `having` expression wraps a stream and applies a boolean operation to each tuple. It emits only tuples for which the boolean operation returns *true*.
-
-=== having Parameters
-
-* `StreamExpression`: (Mandatory) The stream source for the having function.
-* `booleanEvaluator`: (Madatory) The following boolean operations are supported: `eq` (equals), `gt` (greater than), `lt` (less than), `gteq` (greater than or equal to), `lteq` (less than or equal to), `and`, `or`, `eor` (exclusive or), and `not`. Boolean evaluators can be nested with other evaluators to form complex boolean logic.
-
-The comparison evaluators compare the value in a specific field with a value, whether a string, number, or boolean. For example: `eq(field1, 10)`, returns `true` if `field1` is equal to 10.
-
-=== having Syntax
-
-[source,text]
-----
-having(rollup(over=a_s,
- sum(a_i),
- search(collection1,
- q=*:*,
- fl="id,a_s,a_i,a_f",
- sort="a_s asc")),
- and(gt(sum(a_i), 100), lt(sum(a_i), 110)))
-
-----
-
-In this example, the `having` expression iterates the aggregated tuples from the `rollup` expression and emits all tuples where the field `sum(a_i)` is greater then 100 and less then 110.
-
-== leftOuterJoin
-
-The `leftOuterJoin` function wraps two streams, Left and Right, and emits tuples from Left. If there is a tuple in Right equal (as defined by `on`) then the values in that tuple will be included in the emitted tuple. An equal tuple in Right *need not* exist for the Left tuple to be emitted. This supports one-to-one, one-to-many, many-to-one, and many-to-many left outer join scenarios. The tuples are emitted in the order in which they appear in the Left stream. Both streams must be sorted by the fields being used to determine equality (using the `on` parameter). If both tuples contain a field of the same name then the value from the Right stream will be used in the emitted tuple.
-
-You can wrap the incoming streams with a `select` function to be specific about which field values are included in the emitted tuple.
-
-=== leftOuterJoin Parameters
-
-* `StreamExpression for StreamLeft`
-* `StreamExpression for StreamRight`
-* `on`: Fields to be used for checking equality of tuples between Left and Right. Can be of the format `on="fieldName"`, `on="fieldNameInLeft=fieldNameInRight"`, or `on="fieldName, otherFieldName=rightOtherFieldName"`.
-
-=== leftOuterJoin Syntax
-
-[source,text]
-----
-leftOuterJoin(
- search(people, q=*:*, fl="personId,name", sort="personId asc"),
- search(pets, q=type:cat, fl="personId,petName", sort="personId asc"),
- on="personId"
-)
-
-leftOuterJoin(
- search(people, q=*:*, fl="personId,name", sort="personId asc"),
- search(pets, q=type:cat, fl="ownerId,petName", sort="ownerId asc"),
- on="personId=ownerId"
-)
-
-leftOuterJoin(
- search(people, q=*:*, fl="personId,name", sort="personId asc"),
- select(
- search(pets, q=type:cat, fl="ownerId,name", sort="ownerId asc"),
- ownerId,
- name as petName
- ),
- on="personId=ownerId"
-)
-----
-
-== hashJoin
-
-The `hashJoin` function wraps two streams, Left and Right, and for every tuple in Left which exists in Right will emit a tuple containing the fields of both tuples. This supports one-to-one, one-to-many, many-to-one, and many-to-many inner join scenarios. The tuples are emitted in the order in which they appear in the Left stream. The order of the streams does not matter. If both tuples contain a field of the same name then the value from the Right stream will be used in the emitted tuple.
-
-You can wrap the incoming streams with a `select` function to be specific about which field values are included in the emitted tuple.
-
-The hashJoin function can be used when the tuples of Left and Right cannot be put in the same order. Because the tuples are out of order this stream functions by reading all values from the Right stream during the open operation and will store all tuples in memory. The result of this is a memory footprint equal to the size of the Right stream.
-
-=== hashJoin Parameters
-
-* `StreamExpression for StreamLeft`
-* `hashed=StreamExpression for StreamRight`
-* `on`: Fields to be used for checking equality of tuples between Left and Right. Can be of the format `on="fieldName"`, `on="fieldNameInLeft=fieldNameInRight"`, or `on="fieldName, otherFieldName=rightOtherFieldName"`.
-
-=== hashJoin Syntax
-
-[source,text]
-----
-hashJoin(
- search(people, q=*:*, fl="personId,name", sort="personId asc"),
- hashed=search(pets, q=type:cat, fl="personId,petName", sort="personId asc"),
- on="personId"
-)
-
-hashJoin(
- search(people, q=*:*, fl="personId,name", sort="personId asc"),
- hashed=search(pets, q=type:cat, fl="ownerId,petName", sort="ownerId asc"),
- on="personId=ownerId"
-)
-
-hashJoin(
- search(people, q=*:*, fl="personId,name", sort="personId asc"),
- hashed=select(
- search(pets, q=type:cat, fl="ownerId,name", sort="ownerId asc"),
- ownerId,
- name as petName
- ),
- on="personId=ownerId"
-)
-----
-
-== innerJoin
-
-Wraps two streams, Left and Right. For every tuple in Left which exists in Right a tuple containing the fields of both tuples will be emitted. This supports one-to-one, one-to-many, many-to-one, and many-to-many inner join scenarios. The tuples are emitted in the order in which they appear in the Left stream. Both streams must be sorted by the fields being used to determine equality (the 'on' parameter). If both tuples contain a field of the same name then the value from the Right stream will be used in the emitted tuple. You can wrap the incoming streams with a `select(...)` expression to be specific about which field values are included in the emitted tuple.
-
-=== innerJoin Parameters
-
-* `StreamExpression for StreamLeft`
-* `StreamExpression for StreamRight`
-* `on`: Fields to be used for checking equality of tuples between Left and Right. Can be of the format `on="fieldName"`, `on="fieldNameInLeft=fieldNameInRight"`, or `on="fieldName, otherFieldName=rightOtherFieldName"`.
-
-=== innerJoin Syntax
-
-[source,text]
-----
-innerJoin(
- search(people, q=*:*, fl="personId,name", sort="personId asc"),
- search(pets, q=type:cat, fl="personId,petName", sort="personId asc"),
- on="personId"
-)
-
-innerJoin(
- search(people, q=*:*, fl="personId,name", sort="personId asc"),
- search(pets, q=type:cat, fl="ownerId,petName", sort="ownerId asc"),
- on="personId=ownerId"
-)
-
-innerJoin(
- search(people, q=*:*, fl="personId,name", sort="personId asc"),
- select(
- search(pets, q=type:cat, fl="ownerId,name", sort="ownerId asc"),
- ownerId,
- name as petName
- ),
- on="personId=ownerId"
-)
-----
-
-== intersect
-
-The `intersect` function wraps two streams, A and B, and emits tuples from A which *DO* exist in B. The tuples are emitted in the order in which they appear in stream A. Both streams must be sorted by the fields being used to determine equality (the `on` parameter). Only tuples from A are emitted.
-
-=== intersect Parameters
-
-* `StreamExpression for StreamA`
-* `StreamExpression for StreamB`
-* `on`: Fields to be used for checking equality of tuples between A and B. Can be of the format `on="fieldName"`, `on="fieldNameInLeft=fieldNameInRight"`, or `on="fieldName, otherFieldName=rightOtherFieldName"`.
-
-=== intersect Syntax
-
-[source,text]
-----
-intersect(
- search(collection1, q=a_s:(setA || setAB), fl="id,a_s,a_i", sort="a_i asc, a_s asc"),
- search(collection1, q=a_s:(setB || setAB), fl="id,a_s,a_i", sort="a_i asc"),
- on="a_i"
-)
-
-intersect(
- search(collection1, q=a_s:(setA || setAB), fl="id,a_s,a_i", sort="a_i asc, a_s asc"),
- search(collection1, q=a_s:(setB || setAB), fl="id,a_s,a_i", sort="a_i asc, a_s asc"),
- on="a_i,a_s"
-)
-----
-
-== merge
-
-The `merge` function merges two or more streaming expressions and maintains the ordering of the underlying streams. Because the order is maintained, the sorts of the underlying streams must line up with the on parameter provided to the merge function.
-
-=== merge Parameters
-
-* `StreamExpression A`
-* `StreamExpression B`
-* `Optional StreamExpression C,D,....Z`
-* `on`: Sort criteria for performing the merge. Of the form `fieldName order` where order is `asc` or `desc`. Multiple fields can be provided in the form `fieldA order, fieldB order`.
-
-=== merge Syntax
-
-[source,text]
-----
-# Merging two stream expressions together
-merge(
- search(collection1,
- q="id:(0 3 4)",
- fl="id,a_s,a_i,a_f",
- sort="a_f asc"),
- search(collection1,
- q="id:(1)",
- fl="id,a_s,a_i,a_f",
- sort="a_f asc"),
- on="a_f asc")
-----
-
-[source,text]
-----
-# Merging four stream expressions together. Notice that while the sorts of each stream are not identical they are
-# comparable. That is to say the first N fields in each stream's sort matches the N fields in the merge's on clause.
-merge(
- search(collection1,
- q="id:(0 3 4)",
- fl="id,fieldA,fieldB,fieldC",
- sort="fieldA asc, fieldB desc"),
- search(collection1,
- q="id:(1)",
- fl="id,fieldA",
- sort="fieldA asc"),
- search(collection2,
- q="id:(10 11 13)",
- fl="id,fieldA,fieldC",
- sort="fieldA asc"),
- search(collection3,
- q="id:(987)",
- fl="id,fieldA,fieldC",
- sort="fieldA asc"),
- on="fieldA asc")
-----
-
-== null
-
-The null expression is a useful utility function for understanding bottlenecks when performing parallel relational algebra (joins, intersections, rollups etc.). The null function reads all the tuples from an underlying stream and returns a single tuple with the count and processing time. Because the null stream adds minimal overhead of it's own, it can be used to isolate the performance of Solr's /export handler. If the /export handlers performance is not the bottleneck, then the bottleneck is likely occurring in the workers where the stream decorators are running.
-
-The null expression can be wrapped by the parallel function and sent to worker nodes. In this scenario each worker will return one tuple with the count of tuples processed on the worker and the timing information for that worker. This gives valuable information such as:
-
-1. As more workers are added does the performance of the /export handler improve or not.
-2. Are tuples being evenly distributed across the workers, or is the hash partitioning sending more documents to a single worker.
-3. Are all workers processing data at the same speed, or is one of the workers the source of the bottleneck.
-
-=== null Parameters
-
-* `StreamExpression`: (Mandatory) The expression read by the null function.
-
-=== null Syntax
-
-[source,text]
-----
- parallel(workerCollection,
- null(search(collection1, q=*:*, fl="id,a_s,a_i,a_f", sort="a_s desc", qt="/export", partitionKeys="a_s")),
- workers="20",
- zkHost="localhost:9983",
- sort="a_s desc")
-----
-
-The expression above shows a parallel function wrapping a null function. This will cause the null function to be run in parallel across 20 worker nodes. Each worker will return a single tuple with number of tuples processed and time it took to iterate the tuples.
-
-== outerHashJoin
-
-The `outerHashJoin` function wraps two streams, Left and Right, and emits tuples from Left. If there is a tuple in Right equal (as defined by the `on` parameter) then the values in that tuple will be included in the emitted tuple. An equal tuple in Right *need not* exist for the Left tuple to be emitted. This supports one-to-one, one-to-many, many-to-one, and many-to-many left outer join scenarios. The tuples are emitted in the order in which they appear in the Left stream. The order of the streams does not matter. If both tuples contain a field of the same name then the value from the Right stream will be used in the emitted tuple.
-
-You can wrap the incoming streams with a `select` function to be specific about which field values are included in the emitted tuple.
-
-The outerHashJoin stream can be used when the tuples of Left and Right cannot be put in the same order. Because the tuples are out of order, this stream functions by reading all values from the Right stream during the open operation and will store all tuples in memory. The result of this is a memory footprint equal to the size of the Right stream.
-
-=== outerHashJoin Parameters
-
-* `StreamExpression for StreamLeft`
-* `hashed=StreamExpression for StreamRight`
-* `on`: Fields to be used for checking equality of tuples between Left and Right. Can be of the format `on="fieldName"`, `on="fieldNameInLeft=fieldNameInRight"`, or `on="fieldName, otherFieldName=rightOtherFieldName"`.
-
-=== outerHashJoin Syntax
-
-[source,text]
-----
-outerHashJoin(
- search(people, q=*:*, fl="personId,name", sort="personId asc"),
- hashed=search(pets, q=type:cat, fl="personId,petName", sort="personId asc"),
- on="personId"
-)
-
-outerHashJoin(
- search(people, q=*:*, fl="personId,name", sort="personId asc"),
- hashed=search(pets, q=type:cat, fl="ownerId,petName", sort="ownerId asc"),
- on="personId=ownerId"
-)
-
-outerHashJoin(
- search(people, q=*:*, fl="personId,name", sort="personId asc"),
- hashed=select(
- search(pets, q=type:cat, fl="ownerId,name", sort="ownerId asc"),
- ownerId,
- name as petName
- ),
- on="personId=ownerId"
-)
-----
-
-== parallel
-
-The `parallel` function wraps a streaming expression and sends it to N worker nodes to be processed in parallel.
-
-The parallel function requires that the `partitionKeys` parameter be provided to the underlying searches. The `partitionKeys` parameter will partition the search results (tuples) across the worker nodes. Tuples with the same values in the partitionKeys field will be shuffled to the same worker nodes.
-
-The parallel function maintains the sort order of the tuples returned by the worker nodes, so the sort criteria of the parallel function must match up with the sort order of the tuples returned by the workers.
-
-.Worker Collections
-[TIP]
-====
-The worker nodes can be from the same collection as the data, or they can be a different collection entirely, even one that only exists for parallel streaming expressions. A worker collection can be any SolrCloud collection that has the `/stream` handler configured. Unlike normal SolrCloud collections, worker collections don't have to hold any data. Worker collections can be empty collections that exist only to execute streaming expressions.
-====
-
-=== parallel Parameters
-
-* `collection`: Name of the worker collection to send the StreamExpression to.
-* `StreamExpression`: Expression to send to the worker collection.
-* `workers`: Number of workers in the worker collection to send the expression to.
-* `zkHost`: (Optional) The ZooKeeper connect string where the worker collection resides.
-* `sort`: The sort criteria for ordering tuples returned by the worker nodes.
-
-=== parallel Syntax
-
-[source,text]
-----
- parallel(workerCollection,
- reduce(search(collection1, q=*:*, fl="id,a_s,a_i,a_f", sort="a_s desc", partitionKeys="a_s"),
- by="a_s",
- group(sort="a_f desc", n="4")),
- workers="20",
- zkHost="localhost:9983",
- sort="a_s desc")
-----
-
-The expression above shows a `parallel` function wrapping a `reduce` function. This will cause the `reduce` function to be run in parallel across 20 worker nodes.
-
-== priority
-
-The `priority` function is a simple priority scheduler for the <<executor>> function. The `executor` function doesn't directly have a concept of task prioritization; instead it simply executes tasks in the order that they are read from it's underlying stream. The `priority` function provides the ability to schedule a higher priority task ahead of lower priority tasks that were submitted earlier.
-
-The `priority` function wraps two <<stream-sources.adoc#topic,topics>> that are both emitting tuples that contain streaming expressions to execute. The first topic is considered the higher priority task queue.
-
-Each time the `priority` function is called, it checks the higher priority task queue to see if there are any tasks to execute. If tasks are waiting in the higher priority queue then the priority function will emit the higher priority tasks. If there are no high priority tasks to run, the lower priority queue tasks are emitted.
-
-The `priority` function will only emit a batch of tasks from one of the queues each time it is called. This ensures that no lower priority tasks are executed until the higher priority queue has no tasks to run.
-
-=== priority Parameters
-
-* `topic expression`: (Mandatory) the high priority task queue
-* `topic expression`: (Mandatory) the lower priority task queue
-
-=== priority Syntax
-
-[source,text]
-----
-daemon(id="myDaemon",
- executor(threads=10,
- priority(topic(checkpointCollection, storedExpressions, q="priority:high", fl="id, expr_s", initialCheckPoint=0,id="highPriorityTasks"),
- topic(checkpointCollection, storedExpressions, q="priority:low", fl="id, expr_s", initialCheckPoint=0,id="lowPriorityTasks"))))
-----
-
-In the example above the `daemon` function is calling the executor iteratively. Each time it's called, the `executor` function will execute the tasks emitted by the `priority` function. The `priority` function wraps two topics. The first topic is the higher priority task queue, the second topics is the lower priority topic.
-
-== reduce
-
-The `reduce` function wraps an internal stream and groups tuples by common fields.
-
-Each tuple group is operated on as a single block by a pluggable reduce operation. The group operation provided with Solr implements distributed grouping functionality. The group operation also serves as an example reduce operation that can be referred to when building custom reduce operations.
-
-[IMPORTANT]
-====
-The reduce function relies on the sort order of the underlying stream. Accordingly the sort order of the underlying stream must be aligned with the group by field.
-====
-
-=== reduce Parameters
-
-* `StreamExpression`: (Mandatory)
-* `by`: (Mandatory) A comma separated list of fields to group by.
-* `Reduce Operation`: (Mandatory)
-
-=== reduce Syntax
-
-[source,text]
-----
-reduce(search(collection1, q=*:*, fl="id,a_s,a_i,a_f", sort="a_s asc, a_f asc"),
- by="a_s",
- group(sort="a_f desc", n="4")
-)
-----
-
-== rollup
-
-The `rollup` function wraps another stream function and rolls up aggregates over bucket fields. The rollup function relies on the sort order of the underlying stream to rollup aggregates one grouping at a time. Accordingly, the sort order of the underlying stream must match the fields in the `over` parameter of the rollup function.
-
-The rollup function also needs to process entire result sets in order to perform its aggregations. When the underlying stream is the `search` function, the `/export` handler can be used to provide full sorted result sets to the rollup function. This sorted approach allows the rollup function to perform aggregations over very high cardinality fields. The disadvantage of this approach is that the tuples must be sorted and streamed across the network to a worker node to be aggregated. For faster aggregation over low to moderate cardinality fields, the `facet` function can be used.
-
-=== rollup Parameters
-
-* `StreamExpression` (Mandatory)
-* `over`: (Mandatory) A list of fields to group by.
-* `metrics`: (Mandatory) The list of metrics to compute. Currently supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)`, `count(*)`.
-
-=== rollup Syntax
-
-[source,text]
-----
-rollup(
- search(collection1, q=*:*, fl="a_s,a_i,a_f", qt="/export", sort="a_s asc"),
- over="a_s",
- sum(a_i),
- sum(a_f),
- min(a_i),
- min(a_f),
- max(a_i),
- max(a_f),
- avg(a_i),
- avg(a_f),
- count(*)
-)
-----
-
-The example about shows the rollup function wrapping the search function. Notice that search function is using the `/export` handler to provide the entire result set to the rollup stream. Also notice that the search function's *sort param* matches up with the rollup's `over` parameter. This allows the rollup function to rollup the over the `a_s` field, one group at a time.
-
-== scoreNodes
-
-See section in <<graph-traversal.adoc#using-the-scorenodes-function-to-make-a-recommendation,graph traversal>>.
-
-== select
-
-The `select` function wraps a streaming expression and outputs tuples containing a subset or modified set of fields from the incoming tuples. The list of fields included in the output tuple can contain aliases to effectively rename fields. The `select` stream supports both operations and evaluators. One can provide a list of operations and evaluators to perform on any fields, such as `replace, add, if`, etc....
-
-=== select Parameters
-
-* `StreamExpression`
-* `fieldName`: name of field to include in the output tuple (can include multiple of these), such as `outputTuple[fieldName] = inputTuple[fieldName]`
-* `fieldName as aliasFieldName`: aliased field name to include in the output tuple (can include multiple of these), such as `outputTuple[aliasFieldName] = incomingTuple[fieldName]`
-* `replace(fieldName, value, withValue=replacementValue)`: if `incomingTuple[fieldName] == value` then `outgoingTuple[fieldName]` will be set to `replacementValue`. `value` can be the string "null" to replace a null value with some other value.
-* `replace(fieldName, value, withField=otherFieldName)`: if `incomingTuple[fieldName] == value` then `outgoingTuple[fieldName]` will be set to the value of `incomingTuple[otherFieldName]`. `value` can be the string "null" to replace a null value with some other value.
-
-=== select Syntax
-
-[source,text]
-----
-// output tuples with fields teamName, wins, losses, and winPercentages where a null value for wins or losses is translated to the value of 0
-select(
- search(collection1, fl="id,teamName_s,wins,losses", q="*:*", sort="id asc"),
- teamName_s as teamName,
- wins,
- losses,
- replace(wins,null,withValue=0),
- replace(losses,null,withValue=0),
- if(eq(0,wins), 0, div(add(wins,losses), wins)) as winPercentage
-)
-----
-
-== sort
-
-The `sort` function wraps a streaming expression and re-orders the tuples. The sort function emits all incoming tuples in the new sort order. The sort function reads all tuples from the incoming stream, re-orders them using an algorithm with `O(nlog(n))` performance characteristics, where n is the total number of tuples in the incoming stream, and then outputs the tuples in the new sort order. Because all tuples are read into memory, the memory consumption of this function grows linearly with the number of tuples in the incoming stream.
-
-=== sort Parameters
-
-* `StreamExpression`
-* `by`: Sort criteria for re-ordering the tuples
-
-=== sort Syntax
-
-The expression below finds dog owners and orders the results by owner and pet name. Notice that it uses an efficient innerJoin by first ordering by the person/owner id and then re-orders the final output by the owner and pet names.
-
-[source,text]
-----
-sort(
- innerJoin(
- search(people, q=*:*, fl="id,name", sort="id asc"),
- search(pets, q=type:dog, fl="owner,petName", sort="owner asc"),
- on="id=owner"
- ),
- by="name asc, petName asc"
-)
-----
-
-== top
-
-The `top` function wraps a streaming expression and re-orders the tuples. The top function emits only the top N tuples in the new sort order. The top function re-orders the underlying stream so the sort criteria *does not* have to match up with the underlying stream.
-
-=== top Parameters
-
-* `n`: Number of top tuples to return.
-* `StreamExpression`
-* `sort`: Sort criteria for selecting the top N tuples.
-
-=== top Syntax
-
-The expression below finds the top 3 results of the underlying search. Notice that it reverses the sort order. The top function re-orders the results of the underlying stream.
-
-[source,text]
-----
-top(n=3,
- search(collection1,
- q="*:*",
- qt="/export",
- fl="id,a_s,a_i,a_f",
- sort="a_f desc, a_i desc"),
- sort="a_f asc, a_i asc")
-----
-
-== unique
-
-The `unique` function wraps a streaming expression and emits a unique stream of tuples based on the `over` parameter. The unique function relies on the sort order of the underlying stream. The `over` parameter must match up with the sort order of the underlying stream.
-
-The unique function implements a non-co-located unique algorithm. This means that records with the same unique `over` field do not need to be co-located on the same shard. When executed in the parallel, the `partitionKeys` parameter must be the same as the unique `over` field so that records with the same keys will be shuffled to the same worker.
-
-=== unique Parameters
-
-* `StreamExpression`
-* `over`: The unique criteria.
-
-=== unique Syntax
-
-[source,text]
-----
-unique(
- search(collection1,
- q="*:*",
- qt="/export",
- fl="id,a_s,a_i,a_f",
- sort="a_f asc, a_i asc"),
- over="a_f")
-----
-
-== update
-
-The `update` function wraps another functions and sends the tuples to a SolrCloud collection for indexing.
-
-=== update Parameters
-
-* `destinationCollection`: (Mandatory) The collection where the tuples will indexed.
-* `batchSize`: (Mandatory) The indexing batch size.
-* `StreamExpression`: (Mandatory)
-
-=== update Syntax
-
-[source,text]
-----
- update(destinationCollection,
- batchSize=500,
- search(collection1,
- q=*:*,
- fl="id,a_s,a_i,a_f,s_multi,i_multi",
- sort="a_f asc, a_i asc"))
-
-----
-
-The example above sends the tuples returned by the `search` function to the `destinationCollection` to be indexed.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9beafb61/solr/solr-ref-guide/src/stream-evaluator-reference.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/stream-evaluator-reference.adoc b/solr/solr-ref-guide/src/stream-evaluator-reference.adoc
new file mode 100644
index 0000000..eea5ede
--- /dev/null
+++ b/solr/solr-ref-guide/src/stream-evaluator-reference.adoc
@@ -0,0 +1,966 @@
+= Stream Evaluator Reference
+:page-shortname: stream-evaluator-reference
+:page-permalink: stream-evaluator-reference.html
+:page-tocclass: right
+:page-toclevels: 1
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+Stream evaluators are different then stream sources or stream decorators. Both
+stream sources and stream decorators return streams of tuples. Stream evaluators are more like
+a traditional function that evaluates its parameters and
+returns an result. That result can be a single value, array, map or other structure.
+
+Stream evaluators can be nested so that the output of an evaluator becomes the input
+for another evaluator.
+
+Stream evaluators can be called in different contexts. For example a stream evaluator
+can be called on its own or it can be called within the context of a streaming expression.
+
+== abs
+
+The `abs` function will return the absolute value of the provided single parameter. The `abs` function will fail to execute if the value is non-numeric. If a null value is found then null will be returned as the result.
+
+=== abs Parameters
+
+* `Field Name | Raw Number | Number Evaluator`
+
+=== abs Syntax
+
+The expressions below show the various ways in which you can use the `abs` evaluator. Only one parameter is accepted. Returns a numeric value.
+
+[source,text]
+----
+abs(1) // 1, not really a good use case for it
+abs(-1) // 1, not really a good use case for it
+abs(add(fieldA,fieldB)) // absolute value of fieldA + fieldB
+abs(fieldA) // absolute value of fieldA
+----
+
+== acos
+
+The `acos` function returns the trigonometric arccosine of a number.
+
+=== acos Parameters
+
+* `Field Name | Raw Number | Number Evaluator`: The value to return the arccosine of.
+
+=== acos Syntax
+
+[source,text]
+----
+acos(100.4) // returns the arccosine of 100.4
+acos(fieldA) // returns the arccosine for fieldA.
+if(gt(fieldA,fieldB),sin(fieldA),sin(fieldB)) // if fieldA > fieldB then return the arccosine of fieldA, else return the arccosine of fieldB
+----
+
+== add
+
+The `add` function will take 2 or more numeric values and add them together. The `add` function will fail to execute if any of the values are non-numeric. If a null value is found then null will be returned as the result.
+
+=== add Parameters
+
+* `Field Name | Raw Number | Number Evaluator`
+* `Field Name | Raw Number | Number Evaluator`
+* `......`
+* `Field Name | Raw Number | Number Evaluator`
+
+=== add Syntax
+
+The expressions below show the various ways in which you can use the `add` evaluator. The number and order of these parameters do not matter and is not limited except that at least two parameters are required. Returns a numeric value.
+
+[source,text]
+----
+add(1,2,3,4) // 1 + 2 + 3 + 4 == 10
+add(1,fieldA) // 1 + value of fieldA
+add(fieldA,1.4) // value of fieldA + 1.4
+add(fieldA,fieldB,fieldC) // value of fieldA + value of fieldB + value of fieldC
+add(fieldA,div(fieldA,fieldB)) // value of fieldA + (value of fieldA / value of fieldB)
+add(fieldA,if(gt(fieldA,fieldB),fieldA,fieldB)) // if fieldA > fieldB then fieldA + fieldA, else fieldA + fieldB
+----
+
+== analyze
+
+The `analyze` function analyzes text using a Lucene/Solr analyzer and returns a list of tokens
+emitted by the analyzer. The `analyze` function can be called on its own or within the
+`select` and `cartasianProduct` Streaming Expressions.
+
+=== analyze Parameters
+
+* `Field Name` | `Raw Text`: Either the field in a tuple or the raw text to be analyzed.
+* `Analyzer Field Name`: The field name of the analyzer to use to analyze the text.
+
+=== analyze Syntax
+
+The expressions below show the various ways in which you can use the `analyze` evaluator.
+
+* Analyze the raw text: `analyze("hello world", analyzerField)`
+* Analyze a text field within a `select` expression. This will annotate the tuples with output of the analyzer: `select(expr, analyze(textField, analyzerField) as outField)`
+* Analyze a text field with a `cartesianProduct` expression. This will stream each token emitted by the analyzer in its own tuple: `cartesianProduct(expr, analyze(textField, analyzer) as outField)`
+
+== and
+
+The `and` function will return the logical AND of at least 2 boolean parameters. The function will fail to execute if any parameters are non-boolean or null. Returns a boolean value.
+
+=== and Parameters
+
+* `Field Name | Raw Boolean | Boolean Evaluator`
+* `Field Name | Raw Boolean | Boolean Evaluator`
+* `......`
+* `Field Name | Raw Boolean | Boolean Evaluator`
+
+=== and Syntax
+
+The expressions below show the various ways in which you can use the `and` evaluator. At least two parameters are required, but there is no limit to how many you can use.
+
+[source,text]
+----
+and(true,fieldA) // true && fieldA
+and(fieldA,fieldB) // fieldA && fieldB
+and(or(fieldA,fieldB),fieldC) // (fieldA || fieldB) && fieldC
+and(fieldA,fieldB,fieldC,or(fieldD,fieldE),fieldF)
+----
+
+== anova
+
+The `anova` function calculates the https://en.wikipedia.org/wiki/Analysis_of_variance[analysis of variance] for two or more numeric arrays.
+
+=== anova Parameters
+
+//TODO 7.1 - fill in details of Parameters
+* `numeric array` ... (two or more)
+
+=== anova Syntax
+
+[source,text]
+anova(numericArray1, numericArray2) // calculates ANOVA for two numeric arrays
+anova(numericArray1, numericArray2, numericArray2) // calculates ANOVA for three numeric arrays
+
+== array
+
+The `array` function returns an array of numerics or other objects including other arrays.
+
+=== array Parameters
+
+//TODO 7.1 - fill in details of Parameters
+* `numeric` | `array` ...
+
+=== array Syntax
+
+[source,text]
+array(1, 2, 3) // Array of numerics
+array(array(1,2,3), array(4,5,6)) // Array of arrays
+
+== asin
+The `asin` function returns the trigonometric arcsine of a number.
+
+=== asin Parameters
+
+* `Field Name | Raw Number | Number Evaluator`: The value to return the arcsine of.
+
+=== asin Syntax
+
+[source,text]
+----
+asin(100.4) // returns the sine of 100.4
+asine(fieldA) // returns the sine for fieldA.
+if(gt(fieldA,fieldB),asin(fieldA),asin(fieldB)) // if fieldA > fieldB then return the asine of fieldA, else return the asine of fieldB
+----
+
+== atan
+
+The `atan` function returns the trigonometric arctangent of a number.
+
+=== atan Parameters
+
+* `Field Name | Raw Number | Number Evaluator`: The value to return the arctangent of.
+
+=== atan Syntax
+
+[source,text]
+----
+atan(100.4) // returns the arctangent of 100.4
+atan(fieldA) // returns the arctangent for fieldA.
+if(gt(fieldA,fieldB),atan(fieldA),atan(fieldB)) // if fieldA > fieldB then return the arctanget of fieldA, else return the arctangent of fieldB
+----
+
+== cbrt
+
+The `cbrt` function returns the trigonometric cube root of a number.
+
+=== cbrt Parameters
+
+* `Field Name | Raw Number | Number Evaluator`: The value to return the cube root of.
+
+=== cbrt Syntax
+
+[source,text]
+----
+cbrt(100.4) // returns the square root of 100.4
+cbrt(fieldA) // returns the square root for fieldA.
+if(gt(fieldA,fieldB),cbrt(fieldA),cbrt(fieldB)) // if fieldA > fieldB then return the cbrt of fieldA, else return the cbrt of fieldB
+----
+
+== ceil
+The `ceil` function rounds a decimal value to the next highest whole number.
+
+=== ceil Parameters
+
+* `Field Name | Raw Number | Number Evaluator`: The decimal to round up.
+
+=== ceil Syntax
+
+The expressions below show the various ways in which you can use the `ceil` evaluator.
+
+[source,text]
+----
+ceil(100.4) // returns 101.
+ceil(fieldA) // returns the next highest whole number for fieldA.
+if(gt(fieldA,fieldB),ceil(fieldA),ceil(fieldB)) // if fieldA > fieldB then return the ceil of fieldA, else return the ceil of fieldB.
+----
+
+== col
+
+The `col` function returns a numeric array from a list of Tuples. The `col`
+function is used to create numeric arrays from stream sources.
+
+=== col Parameters
+
+//TODO 7.1 - fill in details of Parameters
+* `list of Tuples`
+* `field name`: The field to create the array from.
+
+
+=== col Syntax
+
+[source,text]
+col(tupleList, fieldName)
+
+== conv
+
+The `conv` function returns the https://en.wikipedia.org/wiki/Convolution[convolution] of two numeric arrays.
+
+=== conv Parameters
+
+* `numeric array`
+* `numeric array`
+
+=== conv Syntax
+
+[source,text]
+conv(numericArray1, numericArray2)
+
+== copyOf
+
+The `copyOf` function creates a copy of a numeric array.
+
+=== copyOf Parameters
+
+//TODO 7.1 - fill in details of Parameters
+* `numeric array`
+* `length`: The length of the copied array. The returned array will be right padded with zeros if the length parameter exceeds the size of the original array.
+
+=== copyOf Syntax
+
+[source,text]
+copyOf(numericArray, length)
+
+== copyOfRange
+
+The `copyOfRange` function creates a copy of a range of a numeric array.
+
+=== copyOfRange Parameters
+
+//TODO 7.1 - fill in details of Parameters
+* `numeric array`
+* `start index`
+* `end index`
+
+=== copyOfRange Syntax
+
+[source,text]
+copyOfRange(numericArray, startIndex, endIndex)
+
+== corr
+
+The `corr` function returns the Pearson Product Moment Correlation of two numeric arrays.
+
+=== corr Parameters
+
+//TODO 7.1 - fill in details of Parameters
+* `numeric array`
+* `numeric array`
+
+=== corr Returns
+
+double between -1 and 1
+
+=== corr Synax
+
+[source,text]
+corr(numericArray1, numericArray2)
+
+== cos
+The `cos` function returns the trigonometric cosine of a number.
+
+=== cos Parameters
+
+* `Field Name | Raw Number | Number Evaluator`: The value to return the hyperbolic cosine of.
+
+=== cos Syntax
+
+[source,text]
+----
+cos(100.4) // returns the arccosine of 100.4
+cos(fieldA) // returns the arccosine for fieldA.
+if(gt(fieldA,fieldB),cos(fieldA),cos(fieldB)) // if fieldA > fieldB then return the arccosine of fieldA, else return the cosine of fieldB
+----
+
+== cov
+
+The `cov` function returns the covariance of two numeric arrays.
+
+=== cov Parameters
+
+//TODO 7.1 - fill in details of Parameters
+* `numeric array`
+* `numeric array`
+
+=== cov Syntax
+
+[source,text]
+cov(numericArray, numericArray)
+
+== describe
+
+The `describe` function returns a tuple containing the descriptive statistics for an array.
+
+=== describe Parameters
+
+* `numeric array`
+
+=== describe Syntax
+
+[source,text]
+describe(numericArray)
+
+== distance
+
+The `distance` function calculates the Euclidian distance of two numeric arrays.
+
+=== distance Parameters
+
+* `numeric array`
+* `numeric array`
+
+=== distance Syntax
+
+[source,text]
+distance(numericArray1, numuericArray2))
+
+== div
+
+The `div` function will take two numeric values and divide them. The function will fail to execute if any of the values are non-numeric or null, or the 2nd value is 0. Returns a numeric value.
+
+=== div Parameters
+
+* `Field Name | Raw Number | Number Evaluator`
+* `Field Name | Raw Number | Number Evaluator`
+
+=== div Syntax
+
+The expressions below show the various ways in which you can use the `div` evaluator. The first value will be divided by the second and as such the second cannot be 0.
+
+[source,text]
+----
+div(1,2) // 1 / 2
+div(1,fieldA) // 1 / fieldA
+div(fieldA,1.4) // fieldA / 1.4
+div(fieldA,add(fieldA,fieldB)) // fieldA / (fieldA + fieldB)
+----
+
+== eor
+
+The `eor` function will return the logical exclusive or of at least two boolean parameters. The function will fail to execute if any parameters are non-boolean or null. Returns a boolean value.
+
+=== eor Parameters
+
+* `Field Name | Raw Boolean | Boolean Evaluator`
+* `Field Name | Raw Boolean | Boolean Evaluator`
+* `......`
+* `Field Name | Raw Boolean | Boolean Evaluator`
+
+=== eor Syntax
+
+The expressions below show the various ways in which you can use the `eor` evaluator. At least two parameters are required, but there is no limit to how many you can use.
+
+[source,text]
+----
+eor(true,fieldA) // true iff fieldA is false
+eor(fieldA,fieldB) // true iff either fieldA or fieldB is true but not both
+eor(eq(fieldA,fieldB),eq(fieldC,fieldD)) // true iff either fieldA == fieldB or fieldC == fieldD but not both
+----
+
+== eq
+
+The `eq` function will return whether all the parameters are equal, as per Java's standard `equals(...)` function. The function accepts parameters of any type, but will fail to execute if all the parameters are not of the same type. That is, all are Boolean, all are String, all are Numeric. If any any parameters are null and there is at least one parameter that is not null then false will be returned. Returns a boolean value.
+
+=== eq Parameters
+
+* `Field Name | Raw Value | Evaluator`
+* `Field Name | Raw Value | Evaluator`
+* `......`
+* `Field Name | Raw Value | Evaluator`
+
+=== eq Syntax
+
+The expressions below show the various ways in which you can use the `eq` evaluator.
+
+[source,text]
+----
+eq(1,2) // 1 == 2
+eq(1,fieldA) // 1 == fieldA
+eq(fieldA,val(foo)) fieldA == "foo"
+eq(add(fieldA,fieldB),6) // fieldA + fieldB == 6
+----
+
+== finddelay
+
+The `finddelay` function performs a cross-correlation between two numeric arrays and returns the delay.
+
+=== finddelay Parameters
+
+* `numeric array`
+* `numeric array`
+
+=== finddelay Syntax
+
+[source,text]
+finddelay(numericArray1, numericArray2)
+
+== floor
+The `floor` function rounds a decimal value to the next lowest whole number.
+
+=== floor Parameters
+
+* `Field Name | Raw Number | Number Evaluator`: The decimal to round down.
+
+=== floor Syntax
+
+The expressions below show the various ways in which you can use the `floor` evaluator.
+
+[source,text]
+----
+floor(100.4) // returns 100.
+ceil(fieldA) // returns the next lowestt whole number for fieldA.
+if(gt(fieldA,fieldB),floor(fieldA),floor(fieldB)) // if fieldA > fieldB then return the floor of fieldA, else return the floor of fieldB.
+----
+
+== gt
+
+The `gt` function will return whether the first parameter is greater than the second parameter. The function accepts numeric or string parameters, but will fail to execute if all the parameters are not of the same type. That is, all are String or all are Numeric. If any any parameters are null then an error will be raised. Returns a boolean value.
+
+=== gt Parameters
+
+* `Field Name | Raw Value | Evaluator`
+* `Field Name | Raw Value | Evaluator`
+
+=== gt Syntax
+
+The expressions below show the various ways in which you can use the `gt` evaluator.
+
+[source,text]
+----
+gt(1,2) // 1 > 2
+gt(1,fieldA) // 1 > fieldA
+gt(fieldA,val(foo)) // fieldA > "foo"
+gt(add(fieldA,fieldB),6) // fieldA + fieldB > 6
+----
+
+== gteq
+
+The `gteq` function will return whether the first parameter is greater than or equal to the second parameter. The function accepts numeric and string parameters, but will fail to execute if all the parameters are not of the same type. That is, all are String or all are Numeric. If any any parameters are null then an error will be raised. Returns a boolean value.
+
+=== gteq Parameters
+
+* `Field Name | Raw Value | Evaluator`
+* `Field Name | Raw Value | Evaluator`
+
+=== gteq Syntax
+
+The expressions below show the various ways in which you can use the `gteq` evaluator.
+
+[source,text]
+----
+gteq(1,2) // 1 >= 2
+gteq(1,fieldA) // 1 >= fieldA
+gteq(fieldA,val(foo)) fieldA >= "foo"
+gteq(add(fieldA,fieldB),6) // fieldA + fieldB >= 6
+----
+
+== hist
+
+The `hist` function creates a histogram from a numeric array. The hist function is designed
+to work with continuous variables.
+
+=== hist Parameters
+
+//TODO 7.1 - fill in details of Parameters
+* `numeric array`
+* `bins`: The number of bins in the histogram. Each returned tuple contains
+summary statistics for the observations that were within the bin.
+
+=== hist Sytnax
+
+[source,text]
+hist(numericArray, bins)
+
+== hsin
+The `hsin` function returns the trigonometric hyperbolic sine of a number.
+
+=== hsin Parameters
+
+* `Field Name | Raw Number | Number Evaluator`: The value to return the hyperbolic sine of.
+
+=== hsin Syntax
+
+[source,text]
+----
+hsin(100.4) // returns the hsine of 100.4
+hsin(fieldA) // returns the hsine for fieldA.
+if(gt(fieldA,fieldB),sin(fieldA),sin(fieldB)) // if fieldA > fieldB then return the hsine of fieldA, else return the hsine of fieldB
+----
+
+== if
+
+The `if` function works like a standard conditional if/then statement. If the first parameter is true, then the second parameter will be returned, else the third parameter will be returned. The function accepts a boolean as the first parameter and anything as the second and third parameters. An error will occur if the first parameter is not a boolean or is null.
+
+=== if Parameters
+
+* `Field Name | Raw Value | Boolean Evaluator`
+* `Field Name | Raw Value | Evaluator`
+* `Field Name | Raw Value | Evaluator`
+
+=== if Syntax
+
+The expressions below show the various ways in which you can use the `if` evaluator.
+
+[source,text]
+----
+if(fieldA,fieldB,fieldC) // if fieldA is true then fieldB else fieldC
+if(gt(fieldA,5), fieldA, 5) // if fieldA > 5 then fieldA else 5
+if(eq(fieldB,null), null, div(fieldA,fieldB)) // if fieldB is null then null else fieldA / fieldB
+----
+
+== length
+
+The `length` function returns the length of a numeric array.
+
+=== length Parameters
+
+//TODO 7.1 - fill in details of Parameters
+* `numeric array`
+
+=== length Syntax
+
+[source,text]
+length(numericArray)
+
+== log
+
+The `log` function will return the natural log of the provided single parameter. The `log` function will fail to execute if the value is non-numeric. If a null value is found, then null will be returned as the result.
+
+=== log Parameters
+
+* `Field Name | Raw Number | Number Evaluator`
+
+=== log Syntax
+
+The expressions below show the various ways in which you can use the `log` evaluator. Only one parameter is accepted. Returns a numeric value.
+
+[source,text]
+----
+log(100)
+log(add(fieldA,fieldB))
+log(fieldA)
+----
+
+== lt
+
+The `lt` function will return whether the first parameter is less than the second parameter. The function accepts numeric or string parameters, but will fail to execute if all the parameters are not of the same type. That is, all are String or all are Numeric. If any any parameters are null then an error will be raised. Returns a boolean value.
+
+=== lt Parameters
+
+* `Field Name | Raw Value | Evaluator`
+* `Field Name | Raw Value | Evaluator`
+
+=== lt Syntax
+
+The expressions below show the various ways in which you can use the `lt` evaluator.
+
+[source,text]
+----
+lt(1,2) // 1 < 2
+lt(1,fieldA) // 1 < fieldA
+lt(fieldA,val(foo)) fieldA < "foo"
+lt(add(fieldA,fieldB),6) // fieldA + fieldB < 6
+----
+
+== lteq
+
+The `lteq` function will return whether the first parameter is less than or equal to the second parameter. The function accepts numeric and string parameters, but will fail to execute if all the parameters are not of the same type. That is, all are String or all are Numeric. If any any parameters are null then an error will be raised. Returns a boolean value.
+
+=== lteq Parameters
+
+* `Field Name | Raw Value | Evaluator`
+* `Field Name | Raw Value | Evaluator`
+
+=== lteq Syntax
+
+The expressions below show the various ways in which you can use the `lteq` evaluator.
+
+[source,text]
+----
+lteq(1,2) // 1 <= 2
+lteq(1,fieldA) // 1 <= fieldA
+lteq(fieldA,val(foo)) fieldA <= "foo"
+lteq(add(fieldA,fieldB),6) // fieldA + fieldB <= 6
+----
+
+== mod
+The `mod` function returns the remainder (modulo) of the first parameter divided by the second parameter.
+
+=== mod Parameters
+
+* `Field Name | Raw Number | Number Evaluator`: Parameter 1
+* `Field Name | Raw Number | Number Evaluator`: Parameter 2
+
+=== mod Syntax
+
+The expressions below show the various ways in which you can use the `mod` evaluator.
+
+[source,text]
+----
+mod(100,3) // returns the remainder of 100 / 3 .
+mod(100,fieldA) // returns the remainder of 100 divided by the value of fieldA.
+mod(fieldA,1.4) // returns the remainder of fieldA divided by 1.4.
+if(gt(fieldA,fieldB),mod(fieldA,fieldB),mod(fieldB,fieldA)) // if fieldA > fieldB then return the remainder of fieldA/fieldB, else return the remainder of fieldB/fieldA.
+----
+
+== movingAvg
+
+The `movingAvg` function calculates a https://en.wikipedia.org/wiki/Moving_average[moving average] over an array of numbers.
+
+=== movingAvg Parameters
+
+//TODO 7.1 - fill in details of Parameters
+* `numeric array`
+* `window size`: The array returned will be smaller than this value.
+
+=== movingAvg Syntax
+
+[source,text]
+movingAverage(numericArray, 30)
+
+== mult
+
+The `mult` function will take two or more numeric values and multiply them together. The `mult` function will fail to execute if any of the values are non-numeric. If a null value is found then null will be returned as the result.
+
+=== mult Parameters
+
+* `Field Name | Raw Number | Number Evaluator`
+* `Field Name | Raw Number | Number Evaluator`
+* `......`
+* `Field Name | Raw Number | Number Evaluator`
+
+=== mult Syntax
+
+The expressions below show the various ways in which you can use the `mult` evaluator. The number and order of these parameters do not matter and is not limited except that at least two parameters are required. Returns a numeric value.
+
+[source,text]
+----
+mult(1,2,3,4) // 1 * 2 * 3 * 4
+mult(1,fieldA) // 1 * value of fieldA
+mult(fieldA,1.4) // value of fieldA * 1.4
+mult(fieldA,fieldB,fieldC) // value of fieldA * value of fieldB * value of fieldC
+mult(fieldA,div(fieldA,fieldB)) // value of fieldA * (value of fieldA / value of fieldB)
+mult(fieldA,if(gt(fieldA,fieldB),fieldA,fieldB)) // if fieldA > fieldB then fieldA * fieldA, else fieldA * fieldB
+----
+
+== normalize
+
+The `normalize` function normalizes a numeric array so that values within the array
+have a mean of 0 and standard deviation of 1.
+
+=== normalize Parameters
+
+* `numeric array`
+
+=== normalize Syntax
+
+[source,text]
+normalize(numericArray)
+
+== not
+
+The `not` function will return the logical NOT of a single boolean parameter. The function will fail to execute if the parameter is non-boolean or null. Returns a boolean value.
+
+=== not Parameters
+
+* `Field Name | Raw Boolean | Boolean Evaluator`
+
+=== not Syntax
+
+The expressions below show the various ways in which you can use the `not` evaluator. Only one parameter is allowed.
+
+[source,text]
+----
+not(true) // false
+not(fieldA) // true if fieldA is false else false
+not(eq(fieldA,fieldB)) // true if fieldA != fieldB
+----
+
+== or
+
+The `or` function will return the logical OR of at least 2 boolean parameters. The function will fail to execute if any parameters are non-boolean or null. Returns a boolean value.
+
+=== or Parameters
+
+* `Field Name | Raw Boolean | Boolean Evaluator`
+* `Field Name | Raw Boolean | Boolean Evaluator`
+* `......`
+* `Field Name | Raw Boolean | Boolean Evaluator`
+
+=== or Syntax
+
+The expressions below show the various ways in which you can use the `or` evaluator. At least two parameters are required, but there is no limit to how many you can use.
+
+[source,text]
+----
+or(true,fieldA) // true || fieldA
+or(fieldA,fieldB) // fieldA || fieldB
+or(and(fieldA,fieldB),fieldC) // (fieldA && fieldB) || fieldC
+or(fieldA,fieldB,fieldC,and(fieldD,fieldE),fieldF)
+----
+
+== predict
+
+The `predict` function predicts the value of an dependent variable based on
+the output of the regress function.
+
+=== predict Parameters
+
+//TODO 7.1 - fill in details of Parameters
+* `regress output`
+* `numeric predictor`
+
+=== predict Syntax
+
+[source,text]
+predict(regressOutput, predictor)
+
+== pow
+The `pow` function returns the value of its first parameter raised to the power of its second parameter.
+
+=== pow Parameters
+
+* `Field Name | Raw Number | Number Evaluator`: Parameter 1
+* `Field Name | Raw Number | Number Evaluator`: Parameter 2
+
+=== pow Syntax
+
+The expressions below show the various ways in which you can use the `pow` evaluator.
+
+[source,text]
+----
+pow(2,3) // returns 2 raised to the 3rd power.
+pow(4,fieldA) // returns 4 raised by the value of fieldA.
+pow(fieldA,1.4) // returns the value of fieldA raised by 1.4.
+if(gt(fieldA,fieldB),pow(fieldA,fieldB),pow(fieldB,fieldA)) // if fieldA > fieldB then raise fieldA by fieldB, else raise fieldB by fieldA.
+----
+
+== rank
+
+The `rank` performs a rank transformation on a numeric array.
+
+=== rank Parameters
+
+//TODO 7.1 - fill in details of Parameters
+* `numeric array`
+
+=== rank Syntax
+
+[source,text]
+rank(numericArray)
+
+== raw
+
+The `raw` function will return whatever raw value is the parameter. This is useful for cases where you want to use a string as part of another evaluator.
+
+=== raw Parameters
+
+* `Raw Value`
+
+=== raw Syntax
+
+The expressions below show the various ways in which you can use the `raw` evaluator. Whatever is inside will be returned as-is. Internal evaluators are considered strings and are not evaluated.
+
+[source,text]
+----
+raw(foo) // "foo"
+raw(count(*)) // "count(*)"
+raw(45) // 45
+raw(true) // "true" (note: this returns the string "true" and not the boolean true)
+eq(raw(fieldA), fieldA) // true if the value of fieldA equals the string "fieldA"
+----
+
+== regress
+
+The `regress` function performs a simple regression on two numeric arrays.
+
+The result of this expression is also used by the `predict` function.
+
+=== regress Parameters
+
+//TODO 7.1 - fill in details of Parameters
+* `numeric array`
+* `numeric array`
+
+=== regress Syntax
+
+[source,text]
+regress(numericArray1, numericArray2)
+
+== rev
+
+The `rev` function reverses the order of a numeric array.
+
+=== rev Parameters
+
+* `numeric array`
+
+=== rev Syntax
+
+[source,text]
+rev(numericArray)
+
+== round
+
+The `round` function returns the closest whole number to the argument
+
+=== round Parameters
+
+* `Field Name | Raw Number | Number Evaluator`: The value to return the square root of.
+
+=== round Syntax
+
+[source,text]
+----
+round(100.4)
+round(fieldA)
+if(gt(fieldA,fieldB),sqrt(fieldA),sqrt(fieldB)) // if fieldA > fieldB then return the round of fieldA, else return the round of fieldB
+----
+
+== scale
+
+The `scale` function multiplies all the elements of an array by a number.
+
+=== scale Parameters
+
+//TODO 7.1 - fill in details of Parameters
+* `number`
+* `numeric array`
+
+=== scale Syntax
+
+[source,text]
+scale(number, numericArray)
+
+== sequence
+
+The `sequence` function returns an array of numbers based on its parameters.
+
+=== sequence Parameters
+
+//TODO 7.1 - fill in details of Parameters
+* `length`
+* `start`
+* `stride`
+
+=== sequence Syntax
+
+[source,text]
+sequence(100, 0, 1) // Returns a sequence of length 100, starting from 0 with a stride of 1.
+
+== sin
+The `sin` function returns the trigonometric sine of a number.
+
+=== sin Parameters
+
+* `Field Name | Raw Number | Number Evaluator`: The value to return the sine of.
+
+=== sin Syntax
+
+[source,text]
+----
+sin(100.4) // returns the sine of 100.4
+sine(fieldA) // returns the sine for fieldA.
+if(gt(fieldA,fieldB),sin(fieldA),sin(fieldB)) // if fieldA > fieldB then return the sine of fieldA, else return the sine of fieldB
+----
+
+== sqrt
+
+The `sqrt` function returns the trigonometric square root of a number.
+
+=== sqrt Parameters
+
+* `Field Name | Raw Number | Number Evaluator`: The value to return the square root of.
+
+=== sqrt Syntax
+
+[source,text]
+----
+sqrt(100.4) // returns the square root of 100.4
+sqrt(fieldA) // returns the square root for fieldA.
+if(gt(fieldA,fieldB),sqrt(fieldA),sqrt(fieldB)) // if fieldA > fieldB then return the sqrt of fieldA, else return the sqrt of fieldB
+----
+
+== sub
+
+The `sub` function will take 2 or more numeric values and subtract them, from left to right. The sub function will fail to execute if any of the values are non-numeric. If a null value is found then null will be returned as the result.
+
+=== sub Parameters
+
+* `Field Name | Raw Number | Number Evaluator`
+* `Field Name | Raw Number | Number Evaluator`
+* `......`
+* `Field Name | Raw Number | Number Evaluator`
+
+=== sub Syntax
+
+The expressions below show the various ways in which you can use the `sub` evaluator. The number of these parameters does not matter and is not limited except that at least two parameters are required. Returns a numeric value.
+
+[source,text]
+----
+sub(1,2,3,4) // 1 - 2 - 3 - 4
+sub(1,fieldA) // 1 - value of fieldA
+sub(fieldA,1.4) // value of fieldA - 1.4
+sub(fieldA,fieldB,fieldC) // value of fieldA - value of fieldB - value of fieldC
+sub(fieldA,div(fieldA,fieldB)) // value of fieldA - (value of fieldA / value of fieldB)
+if(gt(fieldA,fieldB),sub(fieldA,fieldB),sub(fieldB,fieldA)) // if fieldA > fieldB then fieldA - fieldB, else fieldB - field
+----
[44/50] [abbrv] lucene-solr:feature/autoscaling:
buildAndPushRelease.py: use wait() instead of poll() to check for process
completion
Posted by no...@apache.org.
buildAndPushRelease.py: use wait() instead of poll() to check for process completion
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/2a13e783
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/2a13e783
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/2a13e783
Branch: refs/heads/feature/autoscaling
Commit: 2a13e783f94be23c3fa2d795e36dd903a7d24585
Parents: 74050a3
Author: Steve Rowe <sa...@gmail.com>
Authored: Mon Oct 2 13:34:30 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Mon Oct 2 13:34:56 2017 -0400
----------------------------------------------------------------------
dev-tools/scripts/buildAndPushRelease.py | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/2a13e783/dev-tools/scripts/buildAndPushRelease.py
----------------------------------------------------------------------
diff --git a/dev-tools/scripts/buildAndPushRelease.py b/dev-tools/scripts/buildAndPushRelease.py
index 1143ebf..cd2fdd1 100644
--- a/dev-tools/scripts/buildAndPushRelease.py
+++ b/dev-tools/scripts/buildAndPushRelease.py
@@ -53,9 +53,14 @@ def runAndSendGPGPassword(command, password):
p.stdin.write((password + '\n').encode('UTF-8'))
p.stdin.write('\n'.encode('UTF-8'))
- result = p.poll()
- if result is not None:
- msg = ' FAILED: %s [see log %s]' % (command, LOG)
+ try:
+ result = p.wait(timeout=120)
+ if result != 0:
+ msg = ' FAILED: %s [see log %s]' % (command, LOG)
+ print(msg)
+ raise RuntimeError(msg)
+ except TimeoutExpired:
+ msg = ' FAILED: %s [timed out after 2 minutes; see log %s]' % (command, LOG)
print(msg)
raise RuntimeError(msg)
[15/50] [abbrv] lucene-solr:feature/autoscaling: LUCENE-7970: Add
test suite for GeoExactCircle. Committed on behalf of Ignacio Vera.
Posted by no...@apache.org.
LUCENE-7970: Add test suite for GeoExactCircle. Committed on behalf of Ignacio Vera.
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/cc16055e
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/cc16055e
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/cc16055e
Branch: refs/heads/feature/autoscaling
Commit: cc16055efe69fa16bdf33a358a44302d30403c64
Parents: ec5704e
Author: Karl Wright <Da...@gmail.com>
Authored: Wed Sep 27 04:31:18 2017 -0400
Committer: Karl Wright <Da...@gmail.com>
Committed: Wed Sep 27 04:31:18 2017 -0400
----------------------------------------------------------------------
.../lucene/spatial3d/geom/GeoCircleFactory.java | 4 +-
.../lucene/spatial3d/geom/GeoExactCircle.java | 176 +++++++------------
.../lucene/spatial3d/geom/GeoCircleTest.java | 20 +++
.../geom/RandomGeo3dShapeGenerator.java | 40 ++++-
4 files changed, 128 insertions(+), 112 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/cc16055e/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoCircleFactory.java
----------------------------------------------------------------------
diff --git a/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoCircleFactory.java b/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoCircleFactory.java
index 7b56e58..8feecce 100644
--- a/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoCircleFactory.java
+++ b/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoCircleFactory.java
@@ -44,8 +44,8 @@ public class GeoCircleFactory {
* @param planetModel is the planet model.
* @param latitude is the center latitude.
* @param longitude is the center longitude.
- * @param radius is the radius angle.
- * @param accuracy is the maximum difference between the circle approximation and the real circle, as computed using
+ * @param radius is the radius surface distance.
+ * @param accuracy is the maximum linear distance between the circle approximation and the real circle, as computed using
* the Vincenty formula.
* @return a GeoCircle corresponding to what was specified.
*/
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/cc16055e/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoExactCircle.java
----------------------------------------------------------------------
diff --git a/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoExactCircle.java b/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoExactCircle.java
index 03eb5ce..e64fd6e 100644
--- a/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoExactCircle.java
+++ b/lucene/spatial3d/src/java/org/apache/lucene/spatial3d/geom/GeoExactCircle.java
@@ -52,8 +52,8 @@ class GeoExactCircle extends GeoBaseCircle {
*@param planetModel is the planet model.
*@param lat is the center latitude.
*@param lon is the center longitude.
- *@param cutoffAngle is the cutoff angle for the circle.
- *@param accuracy is the allowed error value.
+ *@param cutoffAngle is the surface radius for the circle.
+ *@param accuracy is the allowed error value (linear distance).
*/
public GeoExactCircle(final PlanetModel planetModel, final double lat, final double lon, final double cutoffAngle, final double accuracy) {
super(planetModel);
@@ -75,119 +75,76 @@ class GeoExactCircle extends GeoBaseCircle {
actualAccuracy = accuracy;
}
- if (Math.abs(cutoffAngle - Math.PI) < Vector.MINIMUM_RESOLUTION) {
- // Circle is the whole world
- this.circlePlanes = null;
- this.eitherBounds = null;
- this.edgePoints = new GeoPoint[0];
- this.notableEdgePoints = null;
- } else if (planetModel.c == planetModel.ab) {
- // Sphere
- this.eitherBounds = null;
- this.circlePlanes = new ArrayList<>(1);
- // Compute two points on the circle, with the right angle from the center. We'll use these
- // to obtain the perpendicular plane to the circle.
- double upperLat = lat + cutoffAngle;
- double upperLon = lon;
- if (upperLat > Math.PI * 0.5) {
- upperLon += Math.PI;
- if (upperLon > Math.PI)
- upperLon -= 2.0 * Math.PI;
- upperLat = Math.PI - upperLat;
- }
- double lowerLat = lat - cutoffAngle;
- double lowerLon = lon;
- if (lowerLat < -Math.PI * 0.5) {
- lowerLon += Math.PI;
- if (lowerLon > Math.PI)
- lowerLon -= 2.0 * Math.PI;
- lowerLat = -Math.PI - lowerLat;
- }
- final GeoPoint upperPoint = new GeoPoint(planetModel, upperLat, upperLon);
- final GeoPoint lowerPoint = new GeoPoint(planetModel, lowerLat, lowerLon);
- // Construct normal plane
- final Plane normalPlane = Plane.constructNormalizedZPlane(upperPoint, lowerPoint, center);
- // Construct a sided plane that goes through the two points and whose normal is in the normalPlane.
- final SidedPlane circlePlane = SidedPlane.constructNormalizedPerpendicularSidedPlane(center, normalPlane, upperPoint, lowerPoint);
- if (circlePlane == null)
- throw new IllegalArgumentException("Couldn't construct circle plane, probably too small? Cutoff angle = "+cutoffAngle+"; upperPoint = "+upperPoint+"; lowerPoint = "+lowerPoint);
- circlePlanes.add(circlePlane);
- final GeoPoint recomputedIntersectionPoint = circlePlane.getSampleIntersectionPoint(planetModel, normalPlane);
- if (recomputedIntersectionPoint == null)
- throw new IllegalArgumentException("Couldn't construct intersection point, probably circle too small? Plane = "+circlePlane);
- this.edgePoints = new GeoPoint[]{recomputedIntersectionPoint};
- this.notableEdgePoints = null;
- } else {
- this.circlePlanes = new ArrayList<>();
- // If it turns out that there's only one circle plane, this array will be populated but unused
- final List<GeoPoint[]> notableEdgePoints = new ArrayList<>();
-
- // We construct approximation planes until we have a low enough error estimate
- final List<ApproximationSlice> slices = new ArrayList<>(100);
- // Construct four cardinal points, and then we'll build the first two planes
- final GeoPoint northPoint = planetModel.surfacePointOnBearing(center, cutoffAngle, 0.0);
- final GeoPoint southPoint = planetModel.surfacePointOnBearing(center, cutoffAngle, Math.PI);
- final GeoPoint eastPoint = planetModel.surfacePointOnBearing(center, cutoffAngle, Math.PI * 0.5);
- final GeoPoint westPoint = planetModel.surfacePointOnBearing(center, cutoffAngle, Math.PI * 1.5);
+ // Since the provide cutoff angle is really a surface distance, we need to use the point-on-bearing even for spheres.
+ final List<SidedPlane> circlePlanes = new ArrayList<>();
+ // If it turns out that there's only one circle plane, this array will be populated but unused
+ final List<GeoPoint[]> notableEdgePoints = new ArrayList<>();
- this.edgePoints = new GeoPoint[]{northPoint};
+ // We construct approximation planes until we have a low enough error estimate
+ final List<ApproximationSlice> slices = new ArrayList<>(100);
+ // Construct four cardinal points, and then we'll build the first two planes
+ final GeoPoint northPoint = planetModel.surfacePointOnBearing(center, cutoffAngle, 0.0);
+ final GeoPoint southPoint = planetModel.surfacePointOnBearing(center, cutoffAngle, Math.PI);
+ final GeoPoint eastPoint = planetModel.surfacePointOnBearing(center, cutoffAngle, Math.PI * 0.5);
+ final GeoPoint westPoint = planetModel.surfacePointOnBearing(center, cutoffAngle, Math.PI * 1.5);
- if (planetModel.c > planetModel.ab) {
- // z can be greater than x or y, so ellipse is longer in height than width
- slices.add(new ApproximationSlice(center, eastPoint, Math.PI * 0.5, westPoint, Math.PI * -0.5, northPoint, 0.0));
- slices.add(new ApproximationSlice(center, westPoint, Math.PI * 1.5, eastPoint, Math.PI * 0.5, southPoint, Math.PI));
- } else {
- // z will be less than x or y, so ellipse is shorter than it is tall
- slices.add(new ApproximationSlice(center, northPoint, Math.PI * 2.0, southPoint, Math.PI, eastPoint, Math.PI * 0.5));
- slices.add(new ApproximationSlice(center, southPoint, Math.PI, northPoint, 0.0, westPoint, Math.PI * 1.5));
- }
+ if (planetModel.c > planetModel.ab) {
+ // z can be greater than x or y, so ellipse is longer in height than width
+ slices.add(new ApproximationSlice(center, eastPoint, Math.PI * 0.5, westPoint, Math.PI * -0.5, northPoint, 0.0));
+ slices.add(new ApproximationSlice(center, westPoint, Math.PI * 1.5, eastPoint, Math.PI * 0.5, southPoint, Math.PI));
+ } else {
+ // z will be less than x or y, so ellipse is shorter than it is tall
+ slices.add(new ApproximationSlice(center, northPoint, Math.PI * 2.0, southPoint, Math.PI, eastPoint, Math.PI * 0.5));
+ slices.add(new ApproximationSlice(center, southPoint, Math.PI, northPoint, 0.0, westPoint, Math.PI * 1.5));
+ }
- // Now, iterate over slices until we have converted all of them into safe SidedPlanes.
- while (slices.size() > 0) {
- // Peel off a slice from the back
- final ApproximationSlice thisSlice = slices.remove(slices.size()-1);
- // Assess it to see if it is OK as it is, or needs to be split.
- // To do this, we need to look at the part of the circle that will have the greatest error.
- // We will need to compute bearing points for these.
- final double interpPoint1Bearing = (thisSlice.point1Bearing + thisSlice.middlePointBearing) * 0.5;
- final GeoPoint interpPoint1 = planetModel.surfacePointOnBearing(center, cutoffAngle, interpPoint1Bearing);
- final double interpPoint2Bearing = (thisSlice.point2Bearing + thisSlice.middlePointBearing) * 0.5;
- final GeoPoint interpPoint2 = planetModel.surfacePointOnBearing(center, cutoffAngle, interpPoint2Bearing);
- // Is this point on the plane? (that is, is the approximation good enough?)
- if (Math.abs(thisSlice.plane.evaluate(interpPoint1)) < actualAccuracy && Math.abs(thisSlice.plane.evaluate(interpPoint2)) < actualAccuracy) {
- // Good enough; add it to the list of planes, unless it was identical to the previous plane
- if (circlePlanes.size() == 0 || !circlePlanes.get(circlePlanes.size()-1).isNumericallyIdentical(thisSlice.plane)) {
- circlePlanes.add(thisSlice.plane);
- notableEdgePoints.add(new GeoPoint[]{thisSlice.endPoint1, thisSlice.endPoint2});
- }
- } else {
- // Split the plane into two, and add it back to the end
- slices.add(new ApproximationSlice(center,
- thisSlice.endPoint1, thisSlice.point1Bearing,
- thisSlice.middlePoint, thisSlice.middlePointBearing,
- interpPoint1, interpPoint1Bearing));
- slices.add(new ApproximationSlice(center,
- thisSlice.middlePoint, thisSlice.middlePointBearing,
- thisSlice.endPoint2, thisSlice.point2Bearing,
- interpPoint2, interpPoint2Bearing));
+ // Now, iterate over slices until we have converted all of them into safe SidedPlanes.
+ while (slices.size() > 0) {
+ // Peel off a slice from the back
+ final ApproximationSlice thisSlice = slices.remove(slices.size()-1);
+ // Assess it to see if it is OK as it is, or needs to be split.
+ // To do this, we need to look at the part of the circle that will have the greatest error.
+ // We will need to compute bearing points for these.
+ final double interpPoint1Bearing = (thisSlice.point1Bearing + thisSlice.middlePointBearing) * 0.5;
+ final GeoPoint interpPoint1 = planetModel.surfacePointOnBearing(center, cutoffAngle, interpPoint1Bearing);
+ final double interpPoint2Bearing = (thisSlice.point2Bearing + thisSlice.middlePointBearing) * 0.5;
+ final GeoPoint interpPoint2 = planetModel.surfacePointOnBearing(center, cutoffAngle, interpPoint2Bearing);
+ // Is this point on the plane? (that is, is the approximation good enough?)
+ if (Math.abs(thisSlice.plane.evaluate(interpPoint1)) < actualAccuracy && Math.abs(thisSlice.plane.evaluate(interpPoint2)) < actualAccuracy) {
+ // Good enough; add it to the list of planes, unless it was identical to the previous plane
+ if (circlePlanes.size() == 0 || !circlePlanes.get(circlePlanes.size()-1).isNumericallyIdentical(thisSlice.plane)) {
+ circlePlanes.add(thisSlice.plane);
+ notableEdgePoints.add(new GeoPoint[]{thisSlice.endPoint1, thisSlice.endPoint2});
}
+ } else {
+ // Split the plane into two, and add it back to the end
+ slices.add(new ApproximationSlice(center,
+ thisSlice.endPoint1, thisSlice.point1Bearing,
+ thisSlice.middlePoint, thisSlice.middlePointBearing,
+ interpPoint1, interpPoint1Bearing));
+ slices.add(new ApproximationSlice(center,
+ thisSlice.middlePoint, thisSlice.middlePointBearing,
+ thisSlice.endPoint2, thisSlice.point2Bearing,
+ interpPoint2, interpPoint2Bearing));
}
+ }
- //System.out.println("Number of planes needed: "+circlePlanes.size());
+ //System.out.println("Number of planes needed: "+circlePlanes.size());
- // Compute bounds
- if (circlePlanes.size() == 1) {
- this.eitherBounds = null;
- this.notableEdgePoints = null;
- } else {
- this.notableEdgePoints = notableEdgePoints;
- this.eitherBounds = new HashMap<>(circlePlanes.size());
- for (int i = 0; i < circlePlanes.size(); i++) {
- final SidedPlane thisPlane = circlePlanes.get(i);
- final SidedPlane previousPlane = (i == 0)?circlePlanes.get(circlePlanes.size()-1):circlePlanes.get(i-1);
- final SidedPlane nextPlane = (i == circlePlanes.size()-1)?circlePlanes.get(0):circlePlanes.get(i+1);
- eitherBounds.put(thisPlane, new EitherBound(previousPlane, nextPlane));
- }
+ this.edgePoints = new GeoPoint[]{northPoint};
+ this.circlePlanes = circlePlanes;
+ // Compute bounds
+ if (circlePlanes.size() == 1) {
+ this.eitherBounds = null;
+ this.notableEdgePoints = null;
+ } else {
+ this.notableEdgePoints = notableEdgePoints;
+ this.eitherBounds = new HashMap<>(circlePlanes.size());
+ for (int i = 0; i < circlePlanes.size(); i++) {
+ final SidedPlane thisPlane = circlePlanes.get(i);
+ final SidedPlane previousPlane = (i == 0)?circlePlanes.get(circlePlanes.size()-1):circlePlanes.get(i-1);
+ final SidedPlane nextPlane = (i == circlePlanes.size()-1)?circlePlanes.get(0):circlePlanes.get(i+1);
+ eitherBounds.put(thisPlane, new EitherBound(previousPlane, nextPlane));
}
}
}
@@ -406,6 +363,9 @@ class GeoExactCircle extends GeoBaseCircle {
this.middlePointBearing = middlePointBearing;
// Construct the plane going through the three given points
this.plane = SidedPlane.constructNormalizedThreePointSidedPlane(center, endPoint1, endPoint2, middlePoint);
+ if (this.plane == null) {
+ throw new IllegalArgumentException("Either circle is too large to fit on ellipsoid or accuracy is too high; could not construct a plane with endPoint1="+endPoint1+" bearing "+point1Bearing+", endPoint2="+endPoint2+" bearing "+point2Bearing+", middle="+middlePoint+" bearing "+middlePointBearing);
+ }
}
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/cc16055e/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/GeoCircleTest.java
----------------------------------------------------------------------
diff --git a/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/GeoCircleTest.java b/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/GeoCircleTest.java
index 21f8d6a..ad21eda 100755
--- a/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/GeoCircleTest.java
+++ b/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/GeoCircleTest.java
@@ -481,5 +481,25 @@ public class GeoCircleTest extends LuceneTestCase {
double surfaceDistance = planetModel.surfaceDistance(center, point);
assertTrue(planetModel.toString() + " " + Double.toString(surfaceDistance - radius) + " " + Double.toString(radius), surfaceDistance - radius < Vector.MINIMUM_ANGULAR_RESOLUTION);
}
+
+ @Test
+ public void exactCircleLargeTest(){
+ boolean success = true;
+ try {
+ GeoCircle circle = GeoCircleFactory.makeExactGeoCircle(new PlanetModel(0.5, 0.7), 0.25 * Math.PI, 0,0.35 * Math.PI, 1e-12);
+ } catch (IllegalArgumentException e) {
+ success = false;
+ }
+ assertTrue(success);
+ success = false;
+ try {
+ GeoCircle circle = GeoCircleFactory.makeExactGeoCircle(PlanetModel.WGS84, 0.25 * Math.PI, 0,0.9996 * Math.PI, 1e-12);
+ } catch (IllegalArgumentException e) {
+ success = true;
+ }
+ assertTrue(success);
+ }
+
+
}
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/cc16055e/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/RandomGeo3dShapeGenerator.java
----------------------------------------------------------------------
diff --git a/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/RandomGeo3dShapeGenerator.java b/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/RandomGeo3dShapeGenerator.java
index d41bf1b..2c29528 100644
--- a/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/RandomGeo3dShapeGenerator.java
+++ b/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/RandomGeo3dShapeGenerator.java
@@ -55,6 +55,7 @@ public class RandomGeo3dShapeGenerator extends LuceneTestCase {
final protected static int COLLECTION = 8;
final protected static int POINT = 9;
final protected static int LINE = 10;
+ final protected static int EXACT_CIRCLE = 11;
/* Helper shapes for generating constraints whch are just three sided polygons */
final protected static int CONVEX_SIMPLE_POLYGON = 500;
@@ -87,7 +88,7 @@ public class RandomGeo3dShapeGenerator extends LuceneTestCase {
* @return a random generated shape code
*/
public int randomShapeType(){
- return random().nextInt(11);
+ return random().nextInt(12);
}
/**
@@ -100,7 +101,7 @@ public class RandomGeo3dShapeGenerator extends LuceneTestCase {
* @return a random generated polygon code
*/
public int randomGeoAreaShapeType(){
- return random().nextInt(11);
+ return random().nextInt(12);
}
/**
@@ -284,6 +285,9 @@ public class RandomGeo3dShapeGenerator extends LuceneTestCase {
case CONCAVE_SIMPLE_POLYGON: {
return concaveSimplePolygon(planetModel, constraints);
}
+ case EXACT_CIRCLE: {
+ return exactCircle(planetModel, constraints);
+ }
default:
throw new IllegalStateException("Unexpected shape type");
}
@@ -351,6 +355,38 @@ public class RandomGeo3dShapeGenerator extends LuceneTestCase {
}
/**
+ * Method that returns a random generated a GeoCircle under given constraints. Returns
+ * NULL if it cannot build the GeoCircle under the given constraints.
+ *
+ * @param planetModel The planet model.
+ * @param constraints The given constraints.
+ * @return The random generated GeoCircle.
+ */
+ private GeoCircle exactCircle(PlanetModel planetModel , Constraints constraints) {
+ int iterations=0;
+ while (iterations < MAX_SHAPE_ITERATIONS) {
+ iterations++;
+ final GeoPoint center = randomGeoPoint(planetModel, constraints);
+ if (center == null){
+ continue;
+ }
+ final double radius = randomCutoffAngle();
+ final int pow = random().nextInt(10) +3;
+ final double accuracy = random().nextDouble() * Math.pow(10, (-1) * pow);
+ try {
+ GeoCircle circle = GeoCircleFactory.makeExactGeoCircle(planetModel, center.getLatitude(), center.getLongitude(), radius, accuracy);
+ if (!constraints.valid(circle)) {
+ continue;
+ }
+ return circle;
+ } catch (IllegalArgumentException e) {
+ continue;
+ }
+ }
+ return null;
+ }
+
+ /**
* Method that returns a random generated a GeoBBox under given constraints. Returns
* NULL if it cannot build the GeoBBox under the given constraints.
*
[45/50] [abbrv] lucene-solr:feature/autoscaling: Ref Guide: fix bad
line breaks lost during conversion from Confluence
Posted by no...@apache.org.
Ref Guide: fix bad line breaks lost during conversion from Confluence
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/266cc82d
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/266cc82d
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/266cc82d
Branch: refs/heads/feature/autoscaling
Commit: 266cc82df0f422c977107d2bd4d403a58eea3f09
Parents: 2a13e78
Author: Cassandra Targett <ct...@apache.org>
Authored: Mon Oct 2 12:42:21 2017 -0500
Committer: Cassandra Targett <ct...@apache.org>
Committed: Mon Oct 2 12:42:21 2017 -0500
----------------------------------------------------------------------
solr/solr-ref-guide/src/upgrading-a-solr-cluster.adoc | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/266cc82d/solr/solr-ref-guide/src/upgrading-a-solr-cluster.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/upgrading-a-solr-cluster.adoc b/solr/solr-ref-guide/src/upgrading-a-solr-cluster.adoc
index 96ccbca..ea790a8 100644
--- a/solr/solr-ref-guide/src/upgrading-a-solr-cluster.adoc
+++ b/solr/solr-ref-guide/src/upgrading-a-solr-cluster.adoc
@@ -59,9 +59,7 @@ Please follow the instructions to install Solr as a Service on Linux documented
[NOTE]
====
-
If you have a `/var/solr/solr.in.sh` file for your existing Solr install, running the `install_solr_service.sh` script will move this file to its new location: `/etc/default/solr.in.sh` (see https://issues.apache.org/jira/browse/SOLR-8101[SOLR-8101] for more details)
-
====
@@ -69,7 +67,11 @@ If you have a `/var/solr/solr.in.sh` file for your existing Solr install, runnin
Open `/etc/default/solr.in.sh` with a text editor and verify that the following variables are set correctly, or add them bottom of the include file as needed:
-`ZK_HOST=SOLR_HOST=SOLR_PORT=SOLR_HOME=`
+[source]
+ZK_HOST=
+SOLR_HOST=
+SOLR_PORT=
+SOLR_HOME=
Make sure the user you plan to own the Solr process is the owner of the `SOLR_HOME` directory. For instance, if you plan to run Solr as the "solr" user and `SOLR_HOME` is `/var/solr/data`, then you would do: `sudo chown -R solr: /var/solr/data`
@@ -80,7 +82,7 @@ You are now ready to start the upgraded Solr node by doing: `sudo service solr s
=== Step 5: Run Healthcheck
-You should run the Solr *healthcheck* command for all collections that are hosted on the upgraded node before proceeding to upgrade the next node in your cluster. For instance, if the newly upgraded node hosts a replica for the *MyDocuments* collection, then you can run the following command (replace ZK_HOST with the ZooKeeper connection string):
+You should run the Solr *healthcheck* command for all collections that are hosted on the upgraded node before proceeding to upgrade the next node in your cluster. For instance, if the newly upgraded node hosts a replica for the *MyDocuments* collection, then you can run the following command (replace `ZK_HOST` with the ZooKeeper connection string):
[source,bash]
----
[29/50] [abbrv] lucene-solr:feature/autoscaling: no Jira - minor
generics / code constructs fixes in MLT
Posted by no...@apache.org.
no Jira - minor generics / code constructs fixes in MLT
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/f9f6d3a3
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/f9f6d3a3
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/f9f6d3a3
Branch: refs/heads/feature/autoscaling
Commit: f9f6d3a3ac43be3a8bd09a5ff131f99e4b5df145
Parents: f1bb914
Author: Tommaso Teofili <to...@apache.org>
Authored: Fri Sep 29 15:32:27 2017 +0200
Committer: Tommaso Teofili <to...@apache.org>
Committed: Fri Sep 29 15:34:52 2017 +0200
----------------------------------------------------------------------
.../org/apache/lucene/queries/mlt/MoreLikeThis.java | 16 ++++------------
1 file changed, 4 insertions(+), 12 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/f9f6d3a3/lucene/queries/src/java/org/apache/lucene/queries/mlt/MoreLikeThis.java
----------------------------------------------------------------------
diff --git a/lucene/queries/src/java/org/apache/lucene/queries/mlt/MoreLikeThis.java b/lucene/queries/src/java/org/apache/lucene/queries/mlt/MoreLikeThis.java
index ea02af3..9afa687 100644
--- a/lucene/queries/src/java/org/apache/lucene/queries/mlt/MoreLikeThis.java
+++ b/lucene/queries/src/java/org/apache/lucene/queries/mlt/MoreLikeThis.java
@@ -782,11 +782,7 @@ public final class MoreLikeThis {
* @param vector List of terms and their frequencies for a doc/field
*/
private void addTermFrequencies(Map<String, Map<String, Int>> field2termFreqMap, Terms vector, String fieldName) throws IOException {
- Map<String, Int> termFreqMap = field2termFreqMap.get(fieldName);
- if (termFreqMap == null) {
- termFreqMap = new HashMap<>();
- field2termFreqMap.put(fieldName, termFreqMap);
- }
+ Map<String, Int> termFreqMap = field2termFreqMap.computeIfAbsent(fieldName, k -> new HashMap<>());
final TermsEnum termsEnum = vector.iterator();
final CharsRefBuilder spare = new CharsRefBuilder();
BytesRef text;
@@ -823,11 +819,7 @@ public final class MoreLikeThis {
throw new UnsupportedOperationException("To use MoreLikeThis without " +
"term vectors, you must provide an Analyzer");
}
- Map<String, Int> termFreqMap = perFieldTermFrequencies.get(fieldName);
- if (termFreqMap == null) {
- termFreqMap = new HashMap<>();
- perFieldTermFrequencies.put(fieldName, termFreqMap);
- }
+ Map<String, Int> termFreqMap = perFieldTermFrequencies.computeIfAbsent(fieldName, k -> new HashMap<>());
try (TokenStream ts = analyzer.tokenStream(fieldName, r)) {
int tokenCount = 0;
// for every token
@@ -906,7 +898,7 @@ public final class MoreLikeThis {
* @see #retrieveInterestingTerms(java.io.Reader, String)
*/
public String[] retrieveInterestingTerms(int docNum) throws IOException {
- ArrayList<Object> al = new ArrayList<>(maxQueryTerms);
+ ArrayList<String> al = new ArrayList<>(maxQueryTerms);
PriorityQueue<ScoreTerm> pq = retrieveTerms(docNum);
ScoreTerm scoreTerm;
int lim = maxQueryTerms; // have to be careful, retrieveTerms returns all words but that's probably not useful to our caller...
@@ -929,7 +921,7 @@ public final class MoreLikeThis {
* @see #setMaxQueryTerms
*/
public String[] retrieveInterestingTerms(Reader r, String fieldName) throws IOException {
- ArrayList<Object> al = new ArrayList<>(maxQueryTerms);
+ ArrayList<String> al = new ArrayList<>(maxQueryTerms);
PriorityQueue<ScoreTerm> pq = retrieveTerms(r, fieldName);
ScoreTerm scoreTerm;
int lim = maxQueryTerms; // have to be careful, retrieveTerms returns all words but that's probably not useful to our caller...
[32/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11422: Add
probabilities parameter to the enumeratedDistribution Stream Evaluator
Posted by no...@apache.org.
SOLR-11422: Add probabilities parameter to the enumeratedDistribution Stream Evaluator
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/c322e369
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/c322e369
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/c322e369
Branch: refs/heads/feature/autoscaling
Commit: c322e36926576ad8cda1379d070598bdd9b43ea1
Parents: 71eb59e
Author: Joel Bernstein <jb...@apache.org>
Authored: Fri Sep 29 13:10:26 2017 -0400
Committer: Joel Bernstein <jb...@apache.org>
Committed: Fri Sep 29 13:10:26 2017 -0400
----------------------------------------------------------------------
.../eval/EnumeratedDistributionEvaluator.java | 19 ++++++++++++++-----
.../solrj/io/stream/StreamExpressionTest.java | 20 ++++++++++++++++++++
2 files changed, 34 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c322e369/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/EnumeratedDistributionEvaluator.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/EnumeratedDistributionEvaluator.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/EnumeratedDistributionEvaluator.java
index a14e54b..4a8b7f7 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/EnumeratedDistributionEvaluator.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/EnumeratedDistributionEvaluator.java
@@ -25,7 +25,7 @@ import org.apache.commons.math3.distribution.EnumeratedIntegerDistribution;
import org.apache.solr.client.solrj.io.stream.expr.StreamExpression;
import org.apache.solr.client.solrj.io.stream.expr.StreamFactory;
-public class EnumeratedDistributionEvaluator extends RecursiveNumericEvaluator implements OneValueWorker {
+public class EnumeratedDistributionEvaluator extends RecursiveNumericEvaluator implements ManyValueWorker {
private static final long serialVersionUID = 1;
@@ -34,12 +34,21 @@ public class EnumeratedDistributionEvaluator extends RecursiveNumericEvaluator i
}
@Override
- public Object doWork(Object first) throws IOException{
- if(null == first){
+ public Object doWork(Object... values) throws IOException{
+ if(values.length == 0){
throw new IOException(String.format(Locale.ROOT,"Invalid expression %s - null found for the first value",toExpression(constructingFactory)));
}
- int[] samples = ((List)first).stream().mapToInt(value -> ((BigDecimal) value).intValue()).toArray();
- return new EnumeratedIntegerDistribution(samples);
+ if(values.length == 1) {
+ List<Number> first = (List<Number>)values[0];
+ int[] samples = ((List) first).stream().mapToInt(value -> ((BigDecimal) value).intValue()).toArray();
+ return new EnumeratedIntegerDistribution(samples);
+ } else {
+ List<Number> first = (List<Number>)values[0];
+ List<Number> second = (List<Number>)values[1];
+ int[] singletons = ((List) first).stream().mapToInt(value -> ((BigDecimal) value).intValue()).toArray();
+ double[] probs = ((List) second).stream().mapToDouble(value -> ((BigDecimal) value).doubleValue()).toArray();
+ return new EnumeratedIntegerDistribution(singletons, probs);
+ }
}
}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/c322e369/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
index d84120f..9e4c6c3 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
@@ -6480,6 +6480,26 @@ public class StreamExpressionTest extends SolrCloudTestCase {
assertEquals(prob.doubleValue(), 0.1, 0.07);
Number cprob = (Number)tuples.get(0).get("c");
assertEquals(cprob.doubleValue(), 0.5, 0.07);
+
+
+ cexpr = "let(a=sample(enumeratedDistribution(array(1,2,3,4), array(40, 30, 20, 10)), 50000),"+
+ "b=freqTable(a),"+
+ "y=col(b, pct))";
+
+ paramsLoc = new ModifiableSolrParams();
+ paramsLoc.set("expr", cexpr);
+ paramsLoc.set("qt", "/stream");
+ url = cluster.getJettySolrRunners().get(0).getBaseUrl().toString()+"/"+COLLECTIONORALIAS;
+ solrStream = new SolrStream(url, paramsLoc);
+ context = new StreamContext();
+ solrStream.setStreamContext(context);
+ tuples = getTuples(solrStream);
+ assertTrue(tuples.size() == 1);
+ List<Number> freqs = (List<Number>)tuples.get(0).get("y");
+ assertEquals(freqs.get(0).doubleValue(), .40, .03);
+ assertEquals(freqs.get(1).doubleValue(), .30, .03);
+ assertEquals(freqs.get(2).doubleValue(), .20, .03);
+ assertEquals(freqs.get(3).doubleValue(), .10, .03);
}
@Test
[25/50] [abbrv] lucene-solr:feature/autoscaling: SOLR-11418: Allow
comments in Streaming Expressions
Posted by no...@apache.org.
SOLR-11418: Allow comments in Streaming Expressions
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/86fa1cf1
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/86fa1cf1
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/86fa1cf1
Branch: refs/heads/feature/autoscaling
Commit: 86fa1cf11bf45e8ca4e78d653d2ef2ceeb584194
Parents: 3f94f2e
Author: Joel Bernstein <jb...@apache.org>
Authored: Thu Sep 28 21:36:13 2017 -0400
Committer: Joel Bernstein <jb...@apache.org>
Committed: Thu Sep 28 21:36:13 2017 -0400
----------------------------------------------------------------------
.../io/stream/expr/StreamExpressionParser.java | 29 ++++++++++++++++++++
.../solrj/io/stream/StreamExpressionTest.java | 3 +-
2 files changed, 31 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/86fa1cf1/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamExpressionParser.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamExpressionParser.java b/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamExpressionParser.java
index fd5c9d3..7ae797f 100644
--- a/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamExpressionParser.java
+++ b/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/expr/StreamExpressionParser.java
@@ -16,6 +16,8 @@
*/
package org.apache.solr.client.solrj.io.stream.expr;
+import java.io.BufferedReader;
+import java.io.StringReader;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
@@ -34,6 +36,7 @@ public class StreamExpressionParser {
}
public static StreamExpression parse(String clause){
+ clause = stripComments(clause);
StreamExpressionParameter expr = generateStreamExpression(clause);
if(null != expr && expr instanceof StreamExpression){
return (StreamExpression)expr;
@@ -41,6 +44,32 @@ public class StreamExpressionParser {
return null;
}
+
+
+ private static String stripComments(String clause) throws RuntimeException {
+ StringBuilder builder = new StringBuilder();
+ BufferedReader reader = null;
+
+ try {
+ reader = new BufferedReader(new StringReader(clause));
+ String line = null;
+ while ((line = reader.readLine()) != null) {
+ if(line.trim().startsWith("#")) {
+ continue;
+ } else {
+ builder.append(line+'\n');
+ }
+ }
+ }catch(Exception e) {
+ throw new RuntimeException(e);
+ } finally{
+ try {
+ reader.close();
+ } catch (Exception e) {}
+ }
+
+ return builder.toString();
+ }
private static StreamExpressionParameter generateStreamExpression(String clause){
String working = clause.trim();
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/86fa1cf1/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
----------------------------------------------------------------------
diff --git a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
index 6799371..6f11382 100644
--- a/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
+++ b/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/StreamExpressionTest.java
@@ -6630,8 +6630,9 @@ public class StreamExpressionTest extends SolrCloudTestCase {
@Test
public void testGammaDistribution() throws Exception {
- String cexpr = "let(echo=true, " +
+ String cexpr = "#comment\nlet(echo=true, " +
"a=describe(sample(gammaDistribution(1, 10),10000)), " +
+ "\n# commment\n"+
"b=describe(sample(gammaDistribution(3, 10),10000)), " +
"c=describe(sample(gammaDistribution(5, 10),10000))," +
"d=describe(sample(gammaDistribution(7, 10),10000))," +
[37/50] [abbrv] lucene-solr:feature/autoscaling:
addBackcompatIndexes.py: only create moreterms, dvupdates,
and empty indexes on unstable branch
Posted by no...@apache.org.
addBackcompatIndexes.py: only create moreterms, dvupdates, and empty indexes on unstable branch
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/ae5caf3f
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/ae5caf3f
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/ae5caf3f
Branch: refs/heads/feature/autoscaling
Commit: ae5caf3fa67df0abcc6ab5190d0b380f94c2dc40
Parents: 0e8d941
Author: Steve Rowe <sa...@gmail.com>
Authored: Sat Sep 30 23:22:46 2017 -0400
Committer: Steve Rowe <sa...@gmail.com>
Committed: Sat Sep 30 23:22:46 2017 -0400
----------------------------------------------------------------------
dev-tools/scripts/addBackcompatIndexes.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/ae5caf3f/dev-tools/scripts/addBackcompatIndexes.py
----------------------------------------------------------------------
diff --git a/dev-tools/scripts/addBackcompatIndexes.py b/dev-tools/scripts/addBackcompatIndexes.py
index 03e858c..e64dbfc 100644
--- a/dev-tools/scripts/addBackcompatIndexes.py
+++ b/dev-tools/scripts/addBackcompatIndexes.py
@@ -246,7 +246,7 @@ def main():
create_and_add_index(source, 'cfs', c.version, current_version, c.temp_dir)
create_and_add_index(source, 'nocfs', c.version, current_version, c.temp_dir)
create_and_add_index(source, 'sorted', c.version, current_version, c.temp_dir)
- if c.version.minor == 0 and c.version.bugfix == 0:
+ if c.version.minor == 0 and c.version.bugfix == 0 and c.version.major < current_version.major:
create_and_add_index(source, 'moreterms', c.version, current_version, c.temp_dir)
create_and_add_index(source, 'dvupdates', c.version, current_version, c.temp_dir)
create_and_add_index(source, 'emptyIndex', c.version, current_version, c.temp_dir)