You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ct...@apache.org on 2017/09/26 02:24:56 UTC
[2/4] lucene-solr:master: Ref Guide: rename Streaming docs to fix
broken intra-doc links in the PDF due to mismatching anchor names
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9beafb61/solr/solr-ref-guide/src/stream-evaluators.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/stream-evaluators.adoc b/solr/solr-ref-guide/src/stream-evaluators.adoc
deleted file mode 100644
index 3599710..0000000
--- a/solr/solr-ref-guide/src/stream-evaluators.adoc
+++ /dev/null
@@ -1,966 +0,0 @@
-= Stream Evaluator Reference
-:page-shortname: stream-evaluators
-:page-permalink: stream-evaluators.html
-:page-tocclass: right
-:page-toclevels: 1
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-
-Stream evaluators are different then stream sources or stream decorators. Both
-stream sources and stream decorators return streams of tuples. Stream evaluators are more like
-a traditional function that evaluates its parameters and
-returns an result. That result can be a single value, array, map or other structure.
-
-Stream evaluators can be nested so that the output of an evaluator becomes the input
-for another evaluator.
-
-Stream evaluators can be called in different contexts. For example a stream evaluator
-can be called on its own or it can be called within the context of a streaming expression.
-
-== abs
-
-The `abs` function will return the absolute value of the provided single parameter. The `abs` function will fail to execute if the value is non-numeric. If a null value is found then null will be returned as the result.
-
-=== abs Parameters
-
-* `Field Name | Raw Number | Number Evaluator`
-
-=== abs Syntax
-
-The expressions below show the various ways in which you can use the `abs` evaluator. Only one parameter is accepted. Returns a numeric value.
-
-[source,text]
-----
-abs(1) // 1, not really a good use case for it
-abs(-1) // 1, not really a good use case for it
-abs(add(fieldA,fieldB)) // absolute value of fieldA + fieldB
-abs(fieldA) // absolute value of fieldA
-----
-
-== acos
-
-The `acos` function returns the trigonometric arccosine of a number.
-
-=== acos Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the arccosine of.
-
-=== acos Syntax
-
-[source,text]
-----
-acos(100.4) // returns the arccosine of 100.4
-acos(fieldA) // returns the arccosine for fieldA.
-if(gt(fieldA,fieldB),sin(fieldA),sin(fieldB)) // if fieldA > fieldB then return the arccosine of fieldA, else return the arccosine of fieldB
-----
-
-== add
-
-The `add` function will take 2 or more numeric values and add them together. The `add` function will fail to execute if any of the values are non-numeric. If a null value is found then null will be returned as the result.
-
-=== add Parameters
-
-* `Field Name | Raw Number | Number Evaluator`
-* `Field Name | Raw Number | Number Evaluator`
-* `......`
-* `Field Name | Raw Number | Number Evaluator`
-
-=== add Syntax
-
-The expressions below show the various ways in which you can use the `add` evaluator. The number and order of these parameters do not matter and is not limited except that at least two parameters are required. Returns a numeric value.
-
-[source,text]
-----
-add(1,2,3,4) // 1 + 2 + 3 + 4 == 10
-add(1,fieldA) // 1 + value of fieldA
-add(fieldA,1.4) // value of fieldA + 1.4
-add(fieldA,fieldB,fieldC) // value of fieldA + value of fieldB + value of fieldC
-add(fieldA,div(fieldA,fieldB)) // value of fieldA + (value of fieldA / value of fieldB)
-add(fieldA,if(gt(fieldA,fieldB),fieldA,fieldB)) // if fieldA > fieldB then fieldA + fieldA, else fieldA + fieldB
-----
-
-== analyze
-
-The `analyze` function analyzes text using a Lucene/Solr analyzer and returns a list of tokens
-emitted by the analyzer. The `analyze` function can be called on its own or within the
-`select` and `cartasianProduct` Streaming Expressions.
-
-=== analyze Parameters
-
-* `Field Name` | `Raw Text`: Either the field in a tuple or the raw text to be analyzed.
-* `Analyzer Field Name`: The field name of the analyzer to use to analyze the text.
-
-=== analyze Syntax
-
-The expressions below show the various ways in which you can use the `analyze` evaluator.
-
-* Analyze the raw text: `analyze("hello world", analyzerField)`
-* Analyze a text field within a `select` expression. This will annotate the tuples with output of the analyzer: `select(expr, analyze(textField, analyzerField) as outField)`
-* Analyze a text field with a `cartesianProduct` expression. This will stream each token emitted by the analyzer in its own tuple: `cartesianProduct(expr, analyze(textField, analyzer) as outField)`
-
-== and
-
-The `and` function will return the logical AND of at least 2 boolean parameters. The function will fail to execute if any parameters are non-boolean or null. Returns a boolean value.
-
-=== and Parameters
-
-* `Field Name | Raw Boolean | Boolean Evaluator`
-* `Field Name | Raw Boolean | Boolean Evaluator`
-* `......`
-* `Field Name | Raw Boolean | Boolean Evaluator`
-
-=== and Syntax
-
-The expressions below show the various ways in which you can use the `and` evaluator. At least two parameters are required, but there is no limit to how many you can use.
-
-[source,text]
-----
-and(true,fieldA) // true && fieldA
-and(fieldA,fieldB) // fieldA && fieldB
-and(or(fieldA,fieldB),fieldC) // (fieldA || fieldB) && fieldC
-and(fieldA,fieldB,fieldC,or(fieldD,fieldE),fieldF)
-----
-
-== anova
-
-The `anova` function calculates the https://en.wikipedia.org/wiki/Analysis_of_variance[analysis of variance] for two or more numeric arrays.
-
-=== anova Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array` ... (two or more)
-
-=== anova Syntax
-
-[source,text]
-anova(numericArray1, numericArray2) // calculates ANOVA for two numeric arrays
-anova(numericArray1, numericArray2, numericArray2) // calculates ANOVA for three numeric arrays
-
-== array
-
-The `array` function returns an array of numerics or other objects including other arrays.
-
-=== array Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric` | `array` ...
-
-=== array Syntax
-
-[source,text]
-array(1, 2, 3) // Array of numerics
-array(array(1,2,3), array(4,5,6)) // Array of arrays
-
-== asin
-The `asin` function returns the trigonometric arcsine of a number.
-
-=== asin Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the arcsine of.
-
-=== asin Syntax
-
-[source,text]
-----
-asin(100.4) // returns the sine of 100.4
-asine(fieldA) // returns the sine for fieldA.
-if(gt(fieldA,fieldB),asin(fieldA),asin(fieldB)) // if fieldA > fieldB then return the asine of fieldA, else return the asine of fieldB
-----
-
-== atan
-
-The `atan` function returns the trigonometric arctangent of a number.
-
-=== atan Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the arctangent of.
-
-=== atan Syntax
-
-[source,text]
-----
-atan(100.4) // returns the arctangent of 100.4
-atan(fieldA) // returns the arctangent for fieldA.
-if(gt(fieldA,fieldB),atan(fieldA),atan(fieldB)) // if fieldA > fieldB then return the arctanget of fieldA, else return the arctangent of fieldB
-----
-
-== cbrt
-
-The `cbrt` function returns the trigonometric cube root of a number.
-
-=== cbrt Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the cube root of.
-
-=== cbrt Syntax
-
-[source,text]
-----
-cbrt(100.4) // returns the square root of 100.4
-cbrt(fieldA) // returns the square root for fieldA.
-if(gt(fieldA,fieldB),cbrt(fieldA),cbrt(fieldB)) // if fieldA > fieldB then return the cbrt of fieldA, else return the cbrt of fieldB
-----
-
-== ceil
-The `ceil` function rounds a decimal value to the next highest whole number.
-
-=== ceil Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The decimal to round up.
-
-=== ceil Syntax
-
-The expressions below show the various ways in which you can use the `ceil` evaluator.
-
-[source,text]
-----
-ceil(100.4) // returns 101.
-ceil(fieldA) // returns the next highest whole number for fieldA.
-if(gt(fieldA,fieldB),ceil(fieldA),ceil(fieldB)) // if fieldA > fieldB then return the ceil of fieldA, else return the ceil of fieldB.
-----
-
-== col
-
-The `col` function returns a numeric array from a list of Tuples. The `col`
-function is used to create numeric arrays from stream sources.
-
-=== col Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `list of Tuples`
-* `field name`: The field to create the array from.
-
-
-=== col Syntax
-
-[source,text]
-col(tupleList, fieldName)
-
-== conv
-
-The `conv` function returns the https://en.wikipedia.org/wiki/Convolution[convolution] of two numeric arrays.
-
-=== conv Parameters
-
-* `numeric array`
-* `numeric array`
-
-=== conv Syntax
-
-[source,text]
-conv(numericArray1, numericArray2)
-
-== copyOf
-
-The `copyOf` function creates a copy of a numeric array.
-
-=== copyOf Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-* `length`: The length of the copied array. The returned array will be right padded with zeros if the length parameter exceeds the size of the original array.
-
-=== copyOf Syntax
-
-[source,text]
-copyOf(numericArray, length)
-
-== copyOfRange
-
-The `copyOfRange` function creates a copy of a range of a numeric array.
-
-=== copyOfRange Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-* `start index`
-* `end index`
-
-=== copyOfRange Syntax
-
-[source,text]
-copyOfRange(numericArray, startIndex, endIndex)
-
-== corr
-
-The `corr` function returns the Pearson Product Moment Correlation of two numeric arrays.
-
-=== corr Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-* `numeric array`
-
-=== corr Returns
-
-double between -1 and 1
-
-=== corr Synax
-
-[source,text]
-corr(numericArray1, numericArray2)
-
-== cos
-The `cos` function returns the trigonometric cosine of a number.
-
-=== cos Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the hyperbolic cosine of.
-
-=== cos Syntax
-
-[source,text]
-----
-cos(100.4) // returns the arccosine of 100.4
-cos(fieldA) // returns the arccosine for fieldA.
-if(gt(fieldA,fieldB),cos(fieldA),cos(fieldB)) // if fieldA > fieldB then return the arccosine of fieldA, else return the cosine of fieldB
-----
-
-== cov
-
-The `cov` function returns the covariance of two numeric arrays.
-
-=== cov Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-* `numeric array`
-
-=== cov Syntax
-
-[source,text]
-cov(numericArray, numericArray)
-
-== describe
-
-The `describe` function returns a tuple containing the descriptive statistics for an array.
-
-=== describe Parameters
-
-* `numeric array`
-
-=== describe Syntax
-
-[source,text]
-describe(numericArray)
-
-== distance
-
-The `distance` function calculates the Euclidian distance of two numeric arrays.
-
-=== distance Parameters
-
-* `numeric array`
-* `numeric array`
-
-=== distance Syntax
-
-[source,text]
-distance(numericArray1, numuericArray2))
-
-== div
-
-The `div` function will take two numeric values and divide them. The function will fail to execute if any of the values are non-numeric or null, or the 2nd value is 0. Returns a numeric value.
-
-=== div Parameters
-
-* `Field Name | Raw Number | Number Evaluator`
-* `Field Name | Raw Number | Number Evaluator`
-
-=== div Syntax
-
-The expressions below show the various ways in which you can use the `div` evaluator. The first value will be divided by the second and as such the second cannot be 0.
-
-[source,text]
-----
-div(1,2) // 1 / 2
-div(1,fieldA) // 1 / fieldA
-div(fieldA,1.4) // fieldA / 1.4
-div(fieldA,add(fieldA,fieldB)) // fieldA / (fieldA + fieldB)
-----
-
-== eor
-
-The `eor` function will return the logical exclusive or of at least two boolean parameters. The function will fail to execute if any parameters are non-boolean or null. Returns a boolean value.
-
-=== eor Parameters
-
-* `Field Name | Raw Boolean | Boolean Evaluator`
-* `Field Name | Raw Boolean | Boolean Evaluator`
-* `......`
-* `Field Name | Raw Boolean | Boolean Evaluator`
-
-=== eor Syntax
-
-The expressions below show the various ways in which you can use the `eor` evaluator. At least two parameters are required, but there is no limit to how many you can use.
-
-[source,text]
-----
-eor(true,fieldA) // true iff fieldA is false
-eor(fieldA,fieldB) // true iff either fieldA or fieldB is true but not both
-eor(eq(fieldA,fieldB),eq(fieldC,fieldD)) // true iff either fieldA == fieldB or fieldC == fieldD but not both
-----
-
-== eq
-
-The `eq` function will return whether all the parameters are equal, as per Java's standard `equals(...)` function. The function accepts parameters of any type, but will fail to execute if all the parameters are not of the same type. That is, all are Boolean, all are String, all are Numeric. If any any parameters are null and there is at least one parameter that is not null then false will be returned. Returns a boolean value.
-
-=== eq Parameters
-
-* `Field Name | Raw Value | Evaluator`
-* `Field Name | Raw Value | Evaluator`
-* `......`
-* `Field Name | Raw Value | Evaluator`
-
-=== eq Syntax
-
-The expressions below show the various ways in which you can use the `eq` evaluator.
-
-[source,text]
-----
-eq(1,2) // 1 == 2
-eq(1,fieldA) // 1 == fieldA
-eq(fieldA,val(foo)) fieldA == "foo"
-eq(add(fieldA,fieldB),6) // fieldA + fieldB == 6
-----
-
-== finddelay
-
-The `finddelay` function performs a cross-correlation between two numeric arrays and returns the delay.
-
-=== finddelay Parameters
-
-* `numeric array`
-* `numeric array`
-
-=== finddelay Syntax
-
-[source,text]
-finddelay(numericArray1, numericArray2)
-
-== floor
-The `floor` function rounds a decimal value to the next lowest whole number.
-
-=== floor Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The decimal to round down.
-
-=== floor Syntax
-
-The expressions below show the various ways in which you can use the `floor` evaluator.
-
-[source,text]
-----
-floor(100.4) // returns 100.
-ceil(fieldA) // returns the next lowestt whole number for fieldA.
-if(gt(fieldA,fieldB),floor(fieldA),floor(fieldB)) // if fieldA > fieldB then return the floor of fieldA, else return the floor of fieldB.
-----
-
-== gt
-
-The `gt` function will return whether the first parameter is greater than the second parameter. The function accepts numeric or string parameters, but will fail to execute if all the parameters are not of the same type. That is, all are String or all are Numeric. If any any parameters are null then an error will be raised. Returns a boolean value.
-
-=== gt Parameters
-
-* `Field Name | Raw Value | Evaluator`
-* `Field Name | Raw Value | Evaluator`
-
-=== gt Syntax
-
-The expressions below show the various ways in which you can use the `gt` evaluator.
-
-[source,text]
-----
-gt(1,2) // 1 > 2
-gt(1,fieldA) // 1 > fieldA
-gt(fieldA,val(foo)) // fieldA > "foo"
-gt(add(fieldA,fieldB),6) // fieldA + fieldB > 6
-----
-
-== gteq
-
-The `gteq` function will return whether the first parameter is greater than or equal to the second parameter. The function accepts numeric and string parameters, but will fail to execute if all the parameters are not of the same type. That is, all are String or all are Numeric. If any any parameters are null then an error will be raised. Returns a boolean value.
-
-=== gteq Parameters
-
-* `Field Name | Raw Value | Evaluator`
-* `Field Name | Raw Value | Evaluator`
-
-=== gteq Syntax
-
-The expressions below show the various ways in which you can use the `gteq` evaluator.
-
-[source,text]
-----
-gteq(1,2) // 1 >= 2
-gteq(1,fieldA) // 1 >= fieldA
-gteq(fieldA,val(foo)) fieldA >= "foo"
-gteq(add(fieldA,fieldB),6) // fieldA + fieldB >= 6
-----
-
-== hist
-
-The `hist` function creates a histogram from a numeric array. The hist function is designed
-to work with continuous variables.
-
-=== hist Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-* `bins`: The number of bins in the histogram. Each returned tuple contains
-summary statistics for the observations that were within the bin.
-
-=== hist Sytnax
-
-[source,text]
-hist(numericArray, bins)
-
-== hsin
-The `hsin` function returns the trigonometric hyperbolic sine of a number.
-
-=== hsin Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the hyperbolic sine of.
-
-=== hsin Syntax
-
-[source,text]
-----
-hsin(100.4) // returns the hsine of 100.4
-hsin(fieldA) // returns the hsine for fieldA.
-if(gt(fieldA,fieldB),sin(fieldA),sin(fieldB)) // if fieldA > fieldB then return the hsine of fieldA, else return the hsine of fieldB
-----
-
-== if
-
-The `if` function works like a standard conditional if/then statement. If the first parameter is true, then the second parameter will be returned, else the third parameter will be returned. The function accepts a boolean as the first parameter and anything as the second and third parameters. An error will occur if the first parameter is not a boolean or is null.
-
-=== if Parameters
-
-* `Field Name | Raw Value | Boolean Evaluator`
-* `Field Name | Raw Value | Evaluator`
-* `Field Name | Raw Value | Evaluator`
-
-=== if Syntax
-
-The expressions below show the various ways in which you can use the `if` evaluator.
-
-[source,text]
-----
-if(fieldA,fieldB,fieldC) // if fieldA is true then fieldB else fieldC
-if(gt(fieldA,5), fieldA, 5) // if fieldA > 5 then fieldA else 5
-if(eq(fieldB,null), null, div(fieldA,fieldB)) // if fieldB is null then null else fieldA / fieldB
-----
-
-== length
-
-The `length` function returns the length of a numeric array.
-
-=== length Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-
-=== length Syntax
-
-[source,text]
-length(numericArray)
-
-== log
-
-The `log` function will return the natural log of the provided single parameter. The `log` function will fail to execute if the value is non-numeric. If a null value is found, then null will be returned as the result.
-
-=== log Parameters
-
-* `Field Name | Raw Number | Number Evaluator`
-
-=== log Syntax
-
-The expressions below show the various ways in which you can use the `log` evaluator. Only one parameter is accepted. Returns a numeric value.
-
-[source,text]
-----
-log(100)
-log(add(fieldA,fieldB))
-log(fieldA)
-----
-
-== lt
-
-The `lt` function will return whether the first parameter is less than the second parameter. The function accepts numeric or string parameters, but will fail to execute if all the parameters are not of the same type. That is, all are String or all are Numeric. If any any parameters are null then an error will be raised. Returns a boolean value.
-
-=== lt Parameters
-
-* `Field Name | Raw Value | Evaluator`
-* `Field Name | Raw Value | Evaluator`
-
-=== lt Syntax
-
-The expressions below show the various ways in which you can use the `lt` evaluator.
-
-[source,text]
-----
-lt(1,2) // 1 < 2
-lt(1,fieldA) // 1 < fieldA
-lt(fieldA,val(foo)) fieldA < "foo"
-lt(add(fieldA,fieldB),6) // fieldA + fieldB < 6
-----
-
-== lteq
-
-The `lteq` function will return whether the first parameter is less than or equal to the second parameter. The function accepts numeric and string parameters, but will fail to execute if all the parameters are not of the same type. That is, all are String or all are Numeric. If any any parameters are null then an error will be raised. Returns a boolean value.
-
-=== lteq Parameters
-
-* `Field Name | Raw Value | Evaluator`
-* `Field Name | Raw Value | Evaluator`
-
-=== lteq Syntax
-
-The expressions below show the various ways in which you can use the `lteq` evaluator.
-
-[source,text]
-----
-lteq(1,2) // 1 <= 2
-lteq(1,fieldA) // 1 <= fieldA
-lteq(fieldA,val(foo)) fieldA <= "foo"
-lteq(add(fieldA,fieldB),6) // fieldA + fieldB <= 6
-----
-
-== mod
-The `mod` function returns the remainder (modulo) of the first parameter divided by the second parameter.
-
-=== mod Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: Parameter 1
-* `Field Name | Raw Number | Number Evaluator`: Parameter 2
-
-=== mod Syntax
-
-The expressions below show the various ways in which you can use the `mod` evaluator.
-
-[source,text]
-----
-mod(100,3) // returns the remainder of 100 / 3 .
-mod(100,fieldA) // returns the remainder of 100 divided by the value of fieldA.
-mod(fieldA,1.4) // returns the remainder of fieldA divided by 1.4.
-if(gt(fieldA,fieldB),mod(fieldA,fieldB),mod(fieldB,fieldA)) // if fieldA > fieldB then return the remainder of fieldA/fieldB, else return the remainder of fieldB/fieldA.
-----
-
-== movingAvg
-
-The `movingAvg` function calculates a https://en.wikipedia.org/wiki/Moving_average[moving average] over an array of numbers.
-
-=== movingAvg Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-* `window size`: The array returned will be smaller than this value.
-
-=== movingAvg Syntax
-
-[source,text]
-movingAverage(numericArray, 30)
-
-== mult
-
-The `mult` function will take two or more numeric values and multiply them together. The `mult` function will fail to execute if any of the values are non-numeric. If a null value is found then null will be returned as the result.
-
-=== mult Parameters
-
-* `Field Name | Raw Number | Number Evaluator`
-* `Field Name | Raw Number | Number Evaluator`
-* `......`
-* `Field Name | Raw Number | Number Evaluator`
-
-=== mult Syntax
-
-The expressions below show the various ways in which you can use the `mult` evaluator. The number and order of these parameters do not matter and is not limited except that at least two parameters are required. Returns a numeric value.
-
-[source,text]
-----
-mult(1,2,3,4) // 1 * 2 * 3 * 4
-mult(1,fieldA) // 1 * value of fieldA
-mult(fieldA,1.4) // value of fieldA * 1.4
-mult(fieldA,fieldB,fieldC) // value of fieldA * value of fieldB * value of fieldC
-mult(fieldA,div(fieldA,fieldB)) // value of fieldA * (value of fieldA / value of fieldB)
-mult(fieldA,if(gt(fieldA,fieldB),fieldA,fieldB)) // if fieldA > fieldB then fieldA * fieldA, else fieldA * fieldB
-----
-
-== normalize
-
-The `normalize` function normalizes a numeric array so that values within the array
-have a mean of 0 and standard deviation of 1.
-
-=== normalize Parameters
-
-* `numeric array`
-
-=== normalize Syntax
-
-[source,text]
-normalize(numericArray)
-
-== not
-
-The `not` function will return the logical NOT of a single boolean parameter. The function will fail to execute if the parameter is non-boolean or null. Returns a boolean value.
-
-=== not Parameters
-
-* `Field Name | Raw Boolean | Boolean Evaluator`
-
-=== not Syntax
-
-The expressions below show the various ways in which you can use the `not` evaluator. Only one parameter is allowed.
-
-[source,text]
-----
-not(true) // false
-not(fieldA) // true if fieldA is false else false
-not(eq(fieldA,fieldB)) // true if fieldA != fieldB
-----
-
-== or
-
-The `or` function will return the logical OR of at least 2 boolean parameters. The function will fail to execute if any parameters are non-boolean or null. Returns a boolean value.
-
-=== or Parameters
-
-* `Field Name | Raw Boolean | Boolean Evaluator`
-* `Field Name | Raw Boolean | Boolean Evaluator`
-* `......`
-* `Field Name | Raw Boolean | Boolean Evaluator`
-
-=== or Syntax
-
-The expressions below show the various ways in which you can use the `or` evaluator. At least two parameters are required, but there is no limit to how many you can use.
-
-[source,text]
-----
-or(true,fieldA) // true || fieldA
-or(fieldA,fieldB) // fieldA || fieldB
-or(and(fieldA,fieldB),fieldC) // (fieldA && fieldB) || fieldC
-or(fieldA,fieldB,fieldC,and(fieldD,fieldE),fieldF)
-----
-
-== predict
-
-The `predict` function predicts the value of an dependent variable based on
-the output of the regress function.
-
-=== predict Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `regress output`
-* `numeric predictor`
-
-=== predict Syntax
-
-[source,text]
-predict(regressOutput, predictor)
-
-== pow
-The `pow` function returns the value of its first parameter raised to the power of its second parameter.
-
-=== pow Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: Parameter 1
-* `Field Name | Raw Number | Number Evaluator`: Parameter 2
-
-=== pow Syntax
-
-The expressions below show the various ways in which you can use the `pow` evaluator.
-
-[source,text]
-----
-pow(2,3) // returns 2 raised to the 3rd power.
-pow(4,fieldA) // returns 4 raised by the value of fieldA.
-pow(fieldA,1.4) // returns the value of fieldA raised by 1.4.
-if(gt(fieldA,fieldB),pow(fieldA,fieldB),pow(fieldB,fieldA)) // if fieldA > fieldB then raise fieldA by fieldB, else raise fieldB by fieldA.
-----
-
-== rank
-
-The `rank` performs a rank transformation on a numeric array.
-
-=== rank Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-
-=== rank Syntax
-
-[source,text]
-rank(numericArray)
-
-== raw
-
-The `raw` function will return whatever raw value is the parameter. This is useful for cases where you want to use a string as part of another evaluator.
-
-=== raw Parameters
-
-* `Raw Value`
-
-=== raw Syntax
-
-The expressions below show the various ways in which you can use the `raw` evaluator. Whatever is inside will be returned as-is. Internal evaluators are considered strings and are not evaluated.
-
-[source,text]
-----
-raw(foo) // "foo"
-raw(count(*)) // "count(*)"
-raw(45) // 45
-raw(true) // "true" (note: this returns the string "true" and not the boolean true)
-eq(raw(fieldA), fieldA) // true if the value of fieldA equals the string "fieldA"
-----
-
-== regress
-
-The `regress` function performs a simple regression on two numeric arrays.
-
-The result of this expression is also used by the `predict` function.
-
-=== regress Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `numeric array`
-* `numeric array`
-
-=== regress Syntax
-
-[source,text]
-regress(numericArray1, numericArray2)
-
-== rev
-
-The `rev` function reverses the order of a numeric array.
-
-=== rev Parameters
-
-* `numeric array`
-
-=== rev Syntax
-
-[source,text]
-rev(numericArray)
-
-== round
-
-The `round` function returns the closest whole number to the argument
-
-=== round Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the square root of.
-
-=== round Syntax
-
-[source,text]
-----
-round(100.4)
-round(fieldA)
-if(gt(fieldA,fieldB),sqrt(fieldA),sqrt(fieldB)) // if fieldA > fieldB then return the round of fieldA, else return the round of fieldB
-----
-
-== scale
-
-The `scale` function multiplies all the elements of an array by a number.
-
-=== scale Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `number`
-* `numeric array`
-
-=== scale Syntax
-
-[source,text]
-scale(number, numericArray)
-
-== sequence
-
-The `sequence` function returns an array of numbers based on its parameters.
-
-=== sequence Parameters
-
-//TODO 7.1 - fill in details of Parameters
-* `length`
-* `start`
-* `stride`
-
-=== sequence Syntax
-
-[source,text]
-sequence(100, 0, 1) // Returns a sequence of length 100, starting from 0 with a stride of 1.
-
-== sin
-The `sin` function returns the trigonometric sine of a number.
-
-=== sin Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the sine of.
-
-=== sin Syntax
-
-[source,text]
-----
-sin(100.4) // returns the sine of 100.4
-sine(fieldA) // returns the sine for fieldA.
-if(gt(fieldA,fieldB),sin(fieldA),sin(fieldB)) // if fieldA > fieldB then return the sine of fieldA, else return the sine of fieldB
-----
-
-== sqrt
-
-The `sqrt` function returns the trigonometric square root of a number.
-
-=== sqrt Parameters
-
-* `Field Name | Raw Number | Number Evaluator`: The value to return the square root of.
-
-=== sqrt Syntax
-
-[source,text]
-----
-sqrt(100.4) // returns the square root of 100.4
-sqrt(fieldA) // returns the square root for fieldA.
-if(gt(fieldA,fieldB),sqrt(fieldA),sqrt(fieldB)) // if fieldA > fieldB then return the sqrt of fieldA, else return the sqrt of fieldB
-----
-
-== sub
-
-The `sub` function will take 2 or more numeric values and subtract them, from left to right. The sub function will fail to execute if any of the values are non-numeric. If a null value is found then null will be returned as the result.
-
-=== sub Parameters
-
-* `Field Name | Raw Number | Number Evaluator`
-* `Field Name | Raw Number | Number Evaluator`
-* `......`
-* `Field Name | Raw Number | Number Evaluator`
-
-=== sub Syntax
-
-The expressions below show the various ways in which you can use the `sub` evaluator. The number of these parameters does not matter and is not limited except that at least two parameters are required. Returns a numeric value.
-
-[source,text]
-----
-sub(1,2,3,4) // 1 - 2 - 3 - 4
-sub(1,fieldA) // 1 - value of fieldA
-sub(fieldA,1.4) // value of fieldA - 1.4
-sub(fieldA,fieldB,fieldC) // value of fieldA - value of fieldB - value of fieldC
-sub(fieldA,div(fieldA,fieldB)) // value of fieldA - (value of fieldA / value of fieldB)
-if(gt(fieldA,fieldB),sub(fieldA,fieldB),sub(fieldB,fieldA)) // if fieldA > fieldB then fieldA - fieldB, else fieldB - field
-----
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9beafb61/solr/solr-ref-guide/src/stream-source-reference.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/stream-source-reference.adoc b/solr/solr-ref-guide/src/stream-source-reference.adoc
new file mode 100644
index 0000000..cf6e451
--- /dev/null
+++ b/solr/solr-ref-guide/src/stream-source-reference.adoc
@@ -0,0 +1,539 @@
+= Stream Source Reference
+:page-shortname: stream-source-reference
+:page-permalink: stream-source-reference.html
+:page-tocclass: right
+:page-toclevels: 1
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+Put something here to see if it fixes things.
+
+== search
+
+The `search` function searches a SolrCloud collection and emits a stream of tuples that match the query. This is very similar to a standard Solr query, and uses many of the same parameters.
+
+This expression allows you to specify a request hander using the `qt` parameter. By default, the `/select` handler is used. The `/select` handler can be used for simple rapid prototyping of expressions. For production, however, you will most likely want to use the `/export` handler which is designed to `sort` and `export` entire result sets. The `/export` handler is not used by default because it has stricter requirements then the `/select` handler so it's not as easy to get started working with. To read more about the `/export` handler requirements review the section <<exporting-result-sets.adoc#exporting-result-sets,Exporting Result Sets>>.
+
+=== search Parameters
+
+* `collection`: (Mandatory) the collection being searched.
+* `q`: (Mandatory) The query to perform on the Solr index.
+* `fl`: (Mandatory) The list of fields to return.
+* `sort`: (Mandatory) The sort criteria.
+* `zkHost`: Only needs to be defined if the collection being searched is found in a different zkHost than the local stream handler.
+* `qt`: Specifies the query type, or request handler, to use. Set this to `/export` to work with large result sets. The default is `/select`.
+* `rows`: (Mandatory with the `/select` handler) The rows parameter specifies how many rows to return. This parameter is only needed with the `/select` handler (which is the default) since the `/export` handler always returns all rows.
+* `partitionKeys`: Comma delimited list of keys to partition the search results by. To be used with the parallel function for parallelizing operations across worker nodes. See the <<stream-decorator-reference.adoc#parallel,parallel>> function for details.
+
+=== search Syntax
+
+[source,text]
+----
+expr=search(collection1,
+ zkHost="localhost:9983",
+ qt="/export",
+ q="*:*",
+ fl="id,a_s,a_i,a_f",
+ sort="a_f asc, a_i asc")
+----
+
+== jdbc
+
+The `jdbc` function searches a JDBC datasource and emits a stream of tuples representing the JDBC result set. Each row in the result set is translated into a tuple and each tuple contains all the cell values for that row.
+
+=== jdbc Parameters
+
+* `connection`: (Mandatory) JDBC formatted connection string to whatever driver you are using.
+* `sql`: (Mandatory) query to pass off to the JDBC endpoint
+* `sort`: (Mandatory) The sort criteria indicating how the data coming out of the JDBC stream is sorted
+* `driver`: The name of the JDBC driver used for the connection. If provided then the driver class will attempt to be loaded into the JVM. If not provided then it is assumed that the driver is already loaded into the JVM. Some drivers require explicit loading so this option is provided.
+* `[driverProperty]`: One or more properties to pass to the JDBC driver during connection. The format is `propertyName="propertyValue"`. You can provide as many of these properties as you'd like and they will all be passed to the connection.
+
+=== Connections and Drivers
+
+Because some JDBC drivers require explicit loading the `driver` parameter can be used to provide the driver class name. If provided, then during stream construction the driver will be loaded. If the driver cannot be loaded because the class is not found on the classpath, then stream construction will fail.
+
+When the JDBC stream is opened it will validate that a driver can be found for the provided connection string. If a driver cannot be found (because it hasn't been loaded) then the open will fail.
+
+=== Datatypes
+
+Due to the inherent differences in datatypes across JDBC sources the following datatypes are supported. The table indicates what Java type will be used for a given JDBC type. Types marked as requiring conversion will go through a conversion for each value of that type. For performance reasons the cell data types are only considered when the stream is opened as this is when the converters are created.
+
+[width="100%",options="header",]
+|===
+|JDBC Type |Java Type |Requires Conversion
+|String |String |No
+|Short |Long |Yes
+|Integer |Long |Yes
+|Long |Long |No
+|Float |Double |Yes
+|Double |Double |No
+|Boolean |Boolean |No
+|===
+
+=== jdbc Syntax
+
+A basic `jdbc` expression:
+
+[source,text]
+----
+jdbc(
+ connection="jdbc:hsqldb:mem:.",
+ sql="select NAME, ADDRESS, EMAIL, AGE from PEOPLE where AGE > 25 order by AGE, NAME DESC",
+ sort="AGE asc, NAME desc",
+ driver="org.hsqldb.jdbcDriver"
+)
+----
+
+A `jdbc` expression that passes a property to the driver:
+
+[source,text]
+----
+// get_column_name is a property to pass to the hsqldb driver
+jdbc(
+ connection="jdbc:hsqldb:mem:.",
+ sql="select NAME as FIRST_NAME, ADDRESS, EMAIL, AGE from PEOPLE where AGE > 25 order by AGE, NAME DESC",
+ sort="AGE asc, NAME desc",
+ driver="org.hsqldb.jdbcDriver",
+ get_column_name="false"
+)
+----
+
+== echo
+
+The `echo` function returns a single Tuple echoing its text parameter. `Echo` is the simplest stream source designed to provide text
+to a text analyzing stream decorator.
+
+=== echo Syntax
+
+[source,text]
+----
+echo("Hello world")
+----
+
+== facet
+
+The `facet` function provides aggregations that are rolled up over buckets. Under the covers the facet function pushes down the aggregation into the search engine using Solr's JSON Facet API. This provides sub-second performance for many use cases. The facet function is appropriate for use with a low to moderate number of distinct values in the bucket fields. To support high cardinality aggregations see the rollup function.
+
+=== facet Parameters
+
+* `collection`: (Mandatory) Collection the facets will be aggregated from.
+* `q`: (Mandatory) The query to build the aggregations from.
+* `buckets`: (Mandatory) Comma separated list of fields to rollup over. The comma separated list represents the dimensions in a multi-dimensional rollup.
+* `bucketSorts`: Comma separated list of sorts to apply to each dimension in the buckets parameters. Sorts can be on the computed metrics or on the bucket values.
+* `bucketSizeLimit`: The number of buckets to include. This value is applied to each dimension.
+* `metrics`: List of metrics to compute for the buckets. Currently supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)`, `count(*)`.
+
+=== facet Syntax
+
+Example 1:
+
+[source,text]
+----
+facet(collection1,
+ q="*:*",
+ buckets="a_s",
+ bucketSorts="sum(a_i) desc",
+ bucketSizeLimit=100,
+ sum(a_i),
+ sum(a_f),
+ min(a_i),
+ min(a_f),
+ max(a_i),
+ max(a_f),
+ avg(a_i),
+ avg(a_f),
+ count(*))
+----
+
+The example above shows a facet function with rollups over a single bucket, where the buckets are returned in descending order by the calculated value of the `sum(a_i)` metric.
+
+Example 2:
+
+[source,text]
+----
+facet(collection1,
+ q="*:*",
+ buckets="year_i, month_i, day_i",
+ bucketSorts="year_i desc, month_i desc, day_i desc",
+ bucketSizeLimit=100,
+ sum(a_i),
+ sum(a_f),
+ min(a_i),
+ min(a_f),
+ max(a_i),
+ max(a_f),
+ avg(a_i),
+ avg(a_f),
+ count(*))
+----
+
+The example above shows a facet function with rollups over three buckets, where the buckets are returned in descending order by bucket value.
+
+== features
+
+The `features` function extracts the key terms from a text field in a classification training set stored in a SolrCloud collection. It uses an algorithm known as * Information Gain* , to select the important terms from the training set. The `features` function was designed to work specifically with the <<train,train>> function, which uses the extracted features to train a text classifier.
+
+The `features` function is designed to work with a training set that provides both positive and negative examples of a class. It emits a tuple for each feature term that is extracted along with the inverse document frequency (IDF) for the term in the training set.
+
+The `features` function uses a query to select the training set from a collection. The IDF for each selected feature is calculated relative to the training set matching the query. This allows multiple training sets to be stored in the same SolrCloud collection without polluting the IDF across training sets.
+
+=== features Parameters
+
+* `collection`: (Mandatory) The collection that holds the training set
+* `q`: (Mandatory) The query that defines the training set. The IDF for the features will be generated specific to the result set matching the query.
+* `featureSet`: (Mandatory) The name of the feature set. This can be used to retrieve the features if they are stored in a SolrCloud collection.
+* `field`: (Mandatory) The text field to extract the features from.
+* `outcome`: (Mandatory) The field that defines the class, positive or negative
+* `numTerms`: (Mandatory) How many feature terms to extract.
+* `positiveLabel`: (defaults to 1) The value in the outcome field that defines a postive outcome.
+
+=== features Syntax
+
+[source,text]
+----
+features(collection1,
+ q="*:*",
+ featureSet="features1",
+ field="body",
+ outcome="out_i",
+ numTerms=250)
+----
+
+== nodes
+
+The `nodes` function provides breadth-first graph traversal. For details, see the section <<graph-traversal.adoc#graph-traversal,Graph Traversal>>.
+
+== knn
+
+The `knn` function returns the K nearest neighbors for a document based on text similarity. Under the covers the `knn` function
+use the More Like This query parser plugin.
+
+=== knn Parameters
+
+* `collection`: (Mandatory) The collection to perform the search in.
+* `id`: (Mandatory) The id of the source document to begin the knn search from.
+* `qf`: (Mandatory) The query field used to compare documents.
+* `k`: (Mandatory) The number of nearest neighbors to return.
+* `fl`: (Mandatory) The field list to return.
+* `mintf`: (Optional) The minimum numer of occurrences of the term in the source document to be inlcued in the search.
+* `maxtf`: (Optional) The maximum numer of occurrences of the term in the source document to be inlcued in the search.
+* `mindf`: (Optional) The minimum numer of occurrences in the corpus to be inlcued in the search.
+* `maxdf`: (Optional) The maximum numer of occurrences in the corpus to be inlcued in the search.
+* `minwl`: (Optional) The minimum world length of to be inlcued in the search.
+* `maxwl`: (Optional) The maximum world length of to be inlcued in the search.
+
+=== knn Syntax
+
+[source,text]
+----
+knn(collection1,
+ id="doc1",
+ qf="text_field",
+ k="10",
+ fl="id, title",
+ mintf="3",
+ maxdf="10000000")
+----
+
+== model
+
+The `model` function retrieves and caches logistic regression text classification models that are stored in a SolrCloud collection. The `model` function is designed to work with models that are created by the <<train,train function>>, but can also be used to retrieve text classification models trained outside of Solr, as long as they conform to the specified format. After the model is retrieved it can be used by the <<stream-decorator-reference.adoc#classify,classify function>> to classify documents.
+
+A single model tuple is fetched and returned based on the *id* parameter. The model is retrieved by matching the *id* parameter with a model name in the index. If more then one iteration of the named model is stored in the index, the highest iteration is selected.
+
+=== Caching with model
+
+The `model` function has an internal LRU (least-recently-used) cache so models do not have to be retrieved with each invocation of the `model` function. The time to cache for each model ID can be passed as a parameter to the function call. Retrieving a cached model does not reset the time for expiring the model ID in the cache.
+
+=== Model Storage
+
+The storage format of the models in Solr is below. The `train` function outputs the format below so you only need to know schema details if you plan to use the `model` function with logistic regression models trained outside of Solr.
+
+* `name_s` (Single value, String, Stored): The name of the model.
+* `iteration_i` (Single value, Integer, Stored): The iteration number of the model. Solr can store all iterations of the models generated by the train function.
+* `terms_ss` (Multi value, String, Stored: The array of terms/features of the model.
+* `weights_ds` (Multi value, double, Stored): The array of term weights. Each weight corresponds by array index to a term.
+* `idfs_ds` (Multi value, double, Stored): The array of term IDFs (Inverse document frequency). Each IDF corresponds by array index to a term.
+
+=== model Parameters
+
+* `collection`: (Mandatory) The collection where the model is stored.
+* `id`: (Mandatory) The id/name of the model. The model function always returns one model. If there are multiple iterations of the name, the highest iteration is returned.
+* `cacheMillis`: (Optional) The amount of time to cache the model in the LRU cache.
+
+=== model Syntax
+
+[source,text]
+----
+model(modelCollection,
+ id="myModel"
+ cacheMillis="200000")
+----
+
+== random
+
+The `random` function searches a SolrCloud collection and emits a pseudo-random set of results that match the query. Each invocation of random will return a different pseudo-random result set.
+
+=== random Parameters
+
+* `collection`: (Mandatory) The collection the stats will be aggregated from.
+* `q`: (Mandatory) The query to build the aggregations from.
+* `rows`: (Mandatory) The number of pseudo-random results to return.
+* fl: (Mandatory) The field list to return.
+* `fq`: (Optional) Filter query
+
+=== random Syntax
+
+[source,text]
+----
+random(baskets,
+ q="productID:productX",
+ rows="100",
+ fl="basketID")
+----
+
+In the example above the `random` function is searching the baskets collections for all rows where "productID:productX". It will return 100 pseudo-random results. The field list returned is the basketID.
+
+== significantTerms
+
+The `significantTerms` function queries a SolrCloud collection, but instead of returning documents, it returns significant terms found in documents in the result set. The `significantTerms` function scores terms based on how frequently they appear in the result set and how rarely they appear in the entire corpus. The `significantTerms` function emits a tuple for each term which contains the term, the score, the foreground count and the background count. The foreground count is how many documents the term appears in in the result set. The background count is how many documents the term appears in in the entire corpus. The foreground and background counts are global for the collection.
+
+=== significantTerms Parameters
+
+* `collection`: (Mandatory) The collection that the function is run on.
+* `q`: (Mandatory) The query that describes the foreground document set.
+* `field`: (Mandatory) The field to extract the terms from.
+* `limit`: (Optional, Default 20) The max number of terms to return.
+* `minDocFreq`: (Optional, Defaults to 5 documents) The minimum number of documents the term must appear in on a shard. This is a float value. If greater then 1.0 then it's considered the absolute number of documents. If less then 1.0 it's treated as a percentage of documents.
+* `maxDocFreq`: (Optional, Defaults to 30% of documents) The maximum number of documents the term can appear in on a shard. This is a float value. If greater then 1.0 then it's considered the absolute number of documents. If less then 1.0 it's treated as a percentage of documents.
+* `minTermLength`: (Optional, Default 4) The minimum length of the term to be considered significant.
+
+=== significantTerms Syntax
+
+[source,text]
+----
+significantTerms(collection1,
+ q="body:Solr",
+ field="author",
+ limit="50",
+ minDocFreq="10",
+ maxDocFreq=".20",
+ minTermLength="5")
+----
+
+In the example above the `significantTerms` function is querying `collection1` and returning at most 50 significant terms from the `authors` field that appear in 10 or more documents but not more then 20% of the corpus.
+
+== shortestPath
+
+The `shortestPath` function is an implementation of a shortest path graph traversal. The `shortestPath` function performs an iterative breadth-first search through an unweighted graph to find the shortest paths between two nodes in a graph. The `shortestPath` function emits a tuple for each path found. Each tuple emitted will contain a `path` key which points to a `List` of nodeIDs comprising the path.
+
+=== shortestPath Parameters
+
+* `collection`: (Mandatory) The collection that the topic query will be run on.
+* `from`: (Mandatory) The nodeID to start the search from
+* `to`: (Mandatory) The nodeID to end the search at
+* `edge`: (Mandatory) Syntax: `from_field=to_field`. The `from_field` defines which field to search from. The `to_field` defines which field to search to. See example below for a detailed explanation.
+* `threads`: (Optional : Default 6) The number of threads used to perform the partitioned join in the traversal.
+* `partitionSize`: (Optional : Default 250) The number of nodes in each partition of the join.
+* `fq`: (Optional) Filter query
+* `maxDepth`: (Mandatory) Limits to the search to a maximum depth in the graph.
+
+=== shortestPath Syntax
+
+[source,text]
+----
+shortestPath(collection,
+ from="john@company.com",
+ to="jane@company.com",
+ edge="from_address=to_address",
+ threads="6",
+ partitionSize="300",
+ fq="limiting query",
+ maxDepth="4")
+----
+
+The expression above performs a breadth-first search to find the shortest paths in an unweighted, directed graph.
+
+The search starts from the nodeID "\john@company.com" in the `from_address` field and searches for the nodeID "\jane@company.com" in the `to_address` field. This search is performed iteratively until the `maxDepth` has been reached. Each level in the traversal is implemented as a parallel partitioned nested loop join across the entire collection. The `threads` parameter controls the number of threads performing the join at each level, while the `partitionSize` parameter controls the of number of nodes in each join partition. The `maxDepth` parameter controls the number of levels to traverse. `fq` is a limiting query applied to each level in the traversal.
+
+== shuffle
+
+The `shuffle` expression sorts and exports entire result sets. The `shuffle` expression is similar to the `search` expression except that
+under the covers `shuffle` always uses the /export handler. The `shuffle` expression is designed to be combined with the relational algebra
+decorators that require complete, sorted result sets. Shuffled result sets can be partitioned across worker nodes with the parallel
+stream decorator to perform parallel relational algebra. When used in parallel mode the partitionKeys parameter must be provided.
+
+=== shuffle Parameters
+
+* `collection`: (Mandatory) the collection being searched.
+* `q`: (Mandatory) The query to perform on the Solr index.
+* `fl`: (Mandatory) The list of fields to return.
+* `sort`: (Mandatory) The sort criteria.
+* `zkHost`: Only needs to be defined if the collection being searched is found in a different zkHost than the local stream handler.
+* `partitionKeys`: Comma delimited list of keys to partition the search results by. To be used with the parallel function for parallelizing operations across worker nodes. See the <<stream-decorator-reference.adoc#parallel,parallel>> function for details.
+
+=== shuffle Syntax
+
+[source,text]
+----
+shuffle(collection1,
+ q="*:*",
+ fl="id,a_s,a_i,a_f",
+ sort="a_f asc, a_i asc")
+----
+
+== stats
+
+The `stats` function gathers simple aggregations for a search result set. The stats function does not support rollups over buckets, so the stats stream always returns a single tuple with the rolled up stats. Under the covers the stats function pushes down the generation of the stats into the search engine using the StatsComponent. The stats function currently supports the following metrics: `count(*)`, `sum()`, `avg()`, `min()`, and `max()`.
+
+=== stats Parameters
+
+* `collection`: (Mandatory) Collection the stats will be aggregated from.
+* `q`: (Mandatory) The query to build the aggregations from.
+* `metrics`: (Mandatory) The metrics to include in the result tuple. Current supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)` and `count(*)`
+
+=== stats Syntax
+
+[source,text]
+----
+stats(collection1,
+ q=*:*,
+ sum(a_i),
+ sum(a_f),
+ min(a_i),
+ min(a_f),
+ max(a_i),
+ max(a_f),
+ avg(a_i),
+ avg(a_f),
+ count(*))
+----
+
+== timeseries
+
+The `timeseries` function builds a time series aggregation. Under the covers the `timeseries` function uses the
+JSON Facet API as its high performance aggregation engine.
+
+=== timeseries Parameters
+
+* `collection`: (Mandatory) Collection the stats will be aggregated from.
+* `q`: (Mandatory) The query to build the aggregations from.
+* `field`: (Mandatory) The date field for the time series.
+* `start`: (Mandatory) The start of the time series expressed in Solr date or date math syntax.
+* `end`: (Mandatory) The end of the time series expressed in Solr date or date math syntax.
+* `gap`: (Mandatory) The time gap between time series aggregation points expressed in Solr date math syntax.
+* `format`: (Optional) Date template to format the date field in the output tuples. Formatting is performed by Java's SimpleDateFormat class.
+* `metrics`: (Mandatory) The metrics to include in the result tuple. Current supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)` and `count(*)`
+
+=== timeseries Syntax
+
+[source,text]
+----
+timeseries(collection1,
+ q=*:*,
+ field="rec_dt"
+ start="NOW-30DAYS",
+ end="NOW",
+ gap="+1DAY",
+ format="YYYY-MM-dd",
+ sum(a_i),
+ max(a_i),
+ max(a_f),
+ avg(a_i),
+ avg(a_f),
+ count(*))
+----
+
+== train
+
+The `train` function trains a Logistic Regression text classifier on a training set stored in a SolrCloud collection. It uses a parallel iterative, batch Gradient Descent approach to train the model. The training algorithm is embedded inside Solr so with each iteration only the model is streamed across the network.
+
+The `train` function wraps a <<features,features>> function which provides the terms and inverse document frequency (IDF) used to train the model. The `train` function operates over the same training set as the `features` function, which includes both positive and negative examples of the class.
+
+With each iteration the `train` function emits a tuple with the model. The model contains the feature terms, weights, and the confusion matrix for the model. The optimized model can then be used to classify documents based on their feature terms.
+
+=== train Parameters
+
+* `collection`: (Mandatory) Collection that holds the training set
+* `q`: (Mandatory) The query that defines the training set. The IDF for the features will be generated on the
+* `name`: (Mandatory) The name of model. This can be used to retrieve the model if they stored in a Solr Cloud collection.
+* `field`: (Mandatory) The text field to extract the features from.
+* `outcome`: (Mandatory) The field that defines the class, positive or negative
+* `maxIterations`: (Mandatory) How many training iterations to perform.
+* `positiveLabel`: (defaults to 1) The value in the outcome field that defines a positive outcome.
+
+=== train Syntax
+
+[source,text]
+----
+train(collection1,
+ features(collection1, q="*:*", featureSet="first", field="body", outcome="out_i", numTerms=250),
+ q="*:*",
+ name="model1",
+ field="body",
+ outcome="out_i",
+ maxIterations=100)
+----
+
+== topic
+
+The `topic` function provides publish/subscribe messaging capabilities built on top of SolrCloud. The topic function allows users to subscribe to a query. The function then provides one-time delivery of new or updated documents that match the topic query. The initial call to the topic function establishes the checkpoints for the specific topic ID. Subsequent calls to the same topic ID will return documents added or updated after the initial checkpoint. Each run of the topic query updates the checkpoints for the topic ID. Setting the initialCheckpoint parameter to 0 will cause the topic to process all documents in the index that match the topic query.
+
+[WARNING]
+====
+The topic function should be considered in beta until https://issues.apache.org/jira/browse/SOLR-8709[SOLR-8709] is committed and released.
+====
+
+=== topic Parameters
+
+* `checkpointCollection`: (Mandatory) The collection where the topic checkpoints are stored.
+* `collection`: (Mandatory) The collection that the topic query will be run on.
+* `id`: (Mandatory) The unique ID for the topic. The checkpoints will be saved under this id.
+* `q`: (Mandatory) The topic query.
+* `fl`: (Mandatory) The field list returned by the topic function.
+* `initialCheckpoint`: (Optional) Sets the initial Solr `\_version_` number to start reading from the queue. If not set, it defaults to the highest version in the index. Setting to 0 will process all records that match query in the index.
+
+=== topic Syntax
+
+[source,text]
+----
+topic(checkpointCollection,
+ collection,
+ id="uniqueId",
+ q="topic query",
+ fl="id, name, country")
+----
+
+== tuple
+
+The `tuple` function emits a single Tuple with name/value pairs. The values can be set to variables assigned in a `let` expression, literals, Stream Evaluators or
+Stream Expressions. In the case of Stream Evaluators the tuple will output the return value from the evaluator.
+This could be a numeric, list or map. If a value is set to a Stream Expression, the `tuple` function will flatten
+the tuple stream from the Stream Expression into a list of Tuples.
+
+=== tuple Parameters
+
+* name=value pairs
+
+=== tuple Syntax
+
+[source,text]
+----
+tuple(a=add(1,1),
+ b=search(collection1, q="cat:a", fl="a, b, c", sort="a desc"))
+----
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9beafb61/solr/solr-ref-guide/src/stream-sources.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/stream-sources.adoc b/solr/solr-ref-guide/src/stream-sources.adoc
deleted file mode 100644
index 2b59edf..0000000
--- a/solr/solr-ref-guide/src/stream-sources.adoc
+++ /dev/null
@@ -1,538 +0,0 @@
-= Stream Source Reference
-:page-shortname: stream-sources
-:page-permalink: stream-sources.html
-:page-tocclass: right
-:page-toclevels: 1
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-
-== search
-
-The `search` function searches a SolrCloud collection and emits a stream of tuples that match the query. This is very similar to a standard Solr query, and uses many of the same parameters.
-
-This expression allows you to specify a request hander using the `qt` parameter. By default, the `/select` handler is used. The `/select` handler can be used for simple rapid prototyping of expressions. For production, however, you will most likely want to use the `/export` handler which is designed to `sort` and `export` entire result sets. The `/export` handler is not used by default because it has stricter requirements then the `/select` handler so it's not as easy to get started working with. To read more about the `/export` handler requirements review the section <<exporting-result-sets.adoc#exporting-result-sets,Exporting Result Sets>>.
-
-=== search Parameters
-
-* `collection`: (Mandatory) the collection being searched.
-* `q`: (Mandatory) The query to perform on the Solr index.
-* `fl`: (Mandatory) The list of fields to return.
-* `sort`: (Mandatory) The sort criteria.
-* `zkHost`: Only needs to be defined if the collection being searched is found in a different zkHost than the local stream handler.
-* `qt`: Specifies the query type, or request handler, to use. Set this to `/export` to work with large result sets. The default is `/select`.
-* `rows`: (Mandatory with the `/select` handler) The rows parameter specifies how many rows to return. This parameter is only needed with the `/select` handler (which is the default) since the `/export` handler always returns all rows.
-* `partitionKeys`: Comma delimited list of keys to partition the search results by. To be used with the parallel function for parallelizing operations across worker nodes. See the <<stream-decorators.adoc#parallel,parallel>> function for details.
-
-=== search Syntax
-
-[source,text]
-----
-expr=search(collection1,
- zkHost="localhost:9983",
- qt="/export",
- q="*:*",
- fl="id,a_s,a_i,a_f",
- sort="a_f asc, a_i asc")
-----
-
-== jdbc
-
-The `jdbc` function searches a JDBC datasource and emits a stream of tuples representing the JDBC result set. Each row in the result set is translated into a tuple and each tuple contains all the cell values for that row.
-
-=== jdbc Parameters
-
-* `connection`: (Mandatory) JDBC formatted connection string to whatever driver you are using.
-* `sql`: (Mandatory) query to pass off to the JDBC endpoint
-* `sort`: (Mandatory) The sort criteria indicating how the data coming out of the JDBC stream is sorted
-* `driver`: The name of the JDBC driver used for the connection. If provided then the driver class will attempt to be loaded into the JVM. If not provided then it is assumed that the driver is already loaded into the JVM. Some drivers require explicit loading so this option is provided.
-* `[driverProperty]`: One or more properties to pass to the JDBC driver during connection. The format is `propertyName="propertyValue"`. You can provide as many of these properties as you'd like and they will all be passed to the connection.
-
-=== Connections and Drivers
-
-Because some JDBC drivers require explicit loading the `driver` parameter can be used to provide the driver class name. If provided, then during stream construction the driver will be loaded. If the driver cannot be loaded because the class is not found on the classpath, then stream construction will fail.
-
-When the JDBC stream is opened it will validate that a driver can be found for the provided connection string. If a driver cannot be found (because it hasn't been loaded) then the open will fail.
-
-=== Datatypes
-
-Due to the inherent differences in datatypes across JDBC sources the following datatypes are supported. The table indicates what Java type will be used for a given JDBC type. Types marked as requiring conversion will go through a conversion for each value of that type. For performance reasons the cell data types are only considered when the stream is opened as this is when the converters are created.
-
-[width="100%",options="header",]
-|===
-|JDBC Type |Java Type |Requires Conversion
-|String |String |No
-|Short |Long |Yes
-|Integer |Long |Yes
-|Long |Long |No
-|Float |Double |Yes
-|Double |Double |No
-|Boolean |Boolean |No
-|===
-
-=== jdbc Syntax
-
-A basic `jdbc` expression:
-
-[source,text]
-----
-jdbc(
- connection="jdbc:hsqldb:mem:.",
- sql="select NAME, ADDRESS, EMAIL, AGE from PEOPLE where AGE > 25 order by AGE, NAME DESC",
- sort="AGE asc, NAME desc",
- driver="org.hsqldb.jdbcDriver"
-)
-----
-
-A `jdbc` expression that passes a property to the driver:
-
-[source,text]
-----
-// get_column_name is a property to pass to the hsqldb driver
-jdbc(
- connection="jdbc:hsqldb:mem:.",
- sql="select NAME as FIRST_NAME, ADDRESS, EMAIL, AGE from PEOPLE where AGE > 25 order by AGE, NAME DESC",
- sort="AGE asc, NAME desc",
- driver="org.hsqldb.jdbcDriver",
- get_column_name="false"
-)
-----
-
-== echo
-
-The `echo` function returns a single Tuple echoing its text parameter. `Echo` is the simplest stream source designed to provide text
-to a text analyzing stream decorator.
-
-=== echo Syntax
-
-[source,text]
-----
-echo("Hello world")
-----
-
-== facet
-
-The `facet` function provides aggregations that are rolled up over buckets. Under the covers the facet function pushes down the aggregation into the search engine using Solr's JSON Facet API. This provides sub-second performance for many use cases. The facet function is appropriate for use with a low to moderate number of distinct values in the bucket fields. To support high cardinality aggregations see the rollup function.
-
-=== facet Parameters
-
-* `collection`: (Mandatory) Collection the facets will be aggregated from.
-* `q`: (Mandatory) The query to build the aggregations from.
-* `buckets`: (Mandatory) Comma separated list of fields to rollup over. The comma separated list represents the dimensions in a multi-dimensional rollup.
-* `bucketSorts`: Comma separated list of sorts to apply to each dimension in the buckets parameters. Sorts can be on the computed metrics or on the bucket values.
-* `bucketSizeLimit`: The number of buckets to include. This value is applied to each dimension.
-* `metrics`: List of metrics to compute for the buckets. Currently supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)`, `count(*)`.
-
-=== facet Syntax
-
-Example 1:
-
-[source,text]
-----
-facet(collection1,
- q="*:*",
- buckets="a_s",
- bucketSorts="sum(a_i) desc",
- bucketSizeLimit=100,
- sum(a_i),
- sum(a_f),
- min(a_i),
- min(a_f),
- max(a_i),
- max(a_f),
- avg(a_i),
- avg(a_f),
- count(*))
-----
-
-The example above shows a facet function with rollups over a single bucket, where the buckets are returned in descending order by the calculated value of the `sum(a_i)` metric.
-
-Example 2:
-
-[source,text]
-----
-facet(collection1,
- q="*:*",
- buckets="year_i, month_i, day_i",
- bucketSorts="year_i desc, month_i desc, day_i desc",
- bucketSizeLimit=100,
- sum(a_i),
- sum(a_f),
- min(a_i),
- min(a_f),
- max(a_i),
- max(a_f),
- avg(a_i),
- avg(a_f),
- count(*))
-----
-
-The example above shows a facet function with rollups over three buckets, where the buckets are returned in descending order by bucket value.
-
-== features
-
-The `features` function extracts the key terms from a text field in a classification training set stored in a SolrCloud collection. It uses an algorithm known as * Information Gain* , to select the important terms from the training set. The `features` function was designed to work specifically with the <<train,train>> function, which uses the extracted features to train a text classifier.
-
-The `features` function is designed to work with a training set that provides both positive and negative examples of a class. It emits a tuple for each feature term that is extracted along with the inverse document frequency (IDF) for the term in the training set.
-
-The `features` function uses a query to select the training set from a collection. The IDF for each selected feature is calculated relative to the training set matching the query. This allows multiple training sets to be stored in the same SolrCloud collection without polluting the IDF across training sets.
-
-=== features Parameters
-
-* `collection`: (Mandatory) The collection that holds the training set
-* `q`: (Mandatory) The query that defines the training set. The IDF for the features will be generated specific to the result set matching the query.
-* `featureSet`: (Mandatory) The name of the feature set. This can be used to retrieve the features if they are stored in a SolrCloud collection.
-* `field`: (Mandatory) The text field to extract the features from.
-* `outcome`: (Mandatory) The field that defines the class, positive or negative
-* `numTerms`: (Mandatory) How many feature terms to extract.
-* `positiveLabel`: (defaults to 1) The value in the outcome field that defines a postive outcome.
-
-=== features Syntax
-
-[source,text]
-----
-features(collection1,
- q="*:*",
- featureSet="features1",
- field="body",
- outcome="out_i",
- numTerms=250)
-----
-
-== nodes
-
-The `nodes` function provides breadth-first graph traversal. For details, see the section <<graph-traversal.adoc#graph-traversal,Graph Traversal>>.
-
-== knn
-
-The `knn` function returns the K nearest neighbors for a document based on text similarity. Under the covers the `knn` function
-use the More Like This query parser plugin.
-
-=== knn Parameters
-
-* `collection`: (Mandatory) The collection to perform the search in.
-* `id`: (Mandatory) The id of the source document to begin the knn search from.
-* `qf`: (Mandatory) The query field used to compare documents.
-* `k`: (Mandatory) The number of nearest neighbors to return.
-* `fl`: (Mandatory) The field list to return.
-* `mintf`: (Optional) The minimum numer of occurrences of the term in the source document to be inlcued in the search.
-* `maxtf`: (Optional) The maximum numer of occurrences of the term in the source document to be inlcued in the search.
-* `mindf`: (Optional) The minimum numer of occurrences in the corpus to be inlcued in the search.
-* `maxdf`: (Optional) The maximum numer of occurrences in the corpus to be inlcued in the search.
-* `minwl`: (Optional) The minimum world length of to be inlcued in the search.
-* `maxwl`: (Optional) The maximum world length of to be inlcued in the search.
-
-=== knn Syntax
-
-[source,text]
-----
-knn(collection1,
- id="doc1",
- qf="text_field",
- k="10",
- fl="id, title",
- mintf="3",
- maxdf="10000000")
-----
-
-== model
-
-The `model` function retrieves and caches logistic regression text classification models that are stored in a SolrCloud collection. The `model` function is designed to work with models that are created by the <<train,train function>>, but can also be used to retrieve text classification models trained outside of Solr, as long as they conform to the specified format. After the model is retrieved it can be used by the <<stream-decorators.adoc#classify,classify function>> to classify documents.
-
-A single model tuple is fetched and returned based on the *id* parameter. The model is retrieved by matching the *id* parameter with a model name in the index. If more then one iteration of the named model is stored in the index, the highest iteration is selected.
-
-=== Caching with model
-
-The `model` function has an internal LRU (least-recently-used) cache so models do not have to be retrieved with each invocation of the `model` function. The time to cache for each model ID can be passed as a parameter to the function call. Retrieving a cached model does not reset the time for expiring the model ID in the cache.
-
-=== Model Storage
-
-The storage format of the models in Solr is below. The `train` function outputs the format below so you only need to know schema details if you plan to use the `model` function with logistic regression models trained outside of Solr.
-
-* `name_s` (Single value, String, Stored): The name of the model.
-* `iteration_i` (Single value, Integer, Stored): The iteration number of the model. Solr can store all iterations of the models generated by the train function.
-* `terms_ss` (Multi value, String, Stored: The array of terms/features of the model.
-* `weights_ds` (Multi value, double, Stored): The array of term weights. Each weight corresponds by array index to a term.
-* `idfs_ds` (Multi value, double, Stored): The array of term IDFs (Inverse document frequency). Each IDF corresponds by array index to a term.
-
-=== model Parameters
-
-* `collection`: (Mandatory) The collection where the model is stored.
-* `id`: (Mandatory) The id/name of the model. The model function always returns one model. If there are multiple iterations of the name, the highest iteration is returned.
-* `cacheMillis`: (Optional) The amount of time to cache the model in the LRU cache.
-
-=== model Syntax
-
-[source,text]
-----
-model(modelCollection,
- id="myModel"
- cacheMillis="200000")
-----
-
-== random
-
-The `random` function searches a SolrCloud collection and emits a pseudo-random set of results that match the query. Each invocation of random will return a different pseudo-random result set.
-
-=== random Parameters
-
-* `collection`: (Mandatory) The collection the stats will be aggregated from.
-* `q`: (Mandatory) The query to build the aggregations from.
-* `rows`: (Mandatory) The number of pseudo-random results to return.
-* fl: (Mandatory) The field list to return.
-* `fq`: (Optional) Filter query
-
-=== random Syntax
-
-[source,text]
-----
-random(baskets,
- q="productID:productX",
- rows="100",
- fl="basketID")
-----
-
-In the example above the `random` function is searching the baskets collections for all rows where "productID:productX". It will return 100 pseudo-random results. The field list returned is the basketID.
-
-== significantTerms
-
-The `significantTerms` function queries a SolrCloud collection, but instead of returning documents, it returns significant terms found in documents in the result set. The `significantTerms` function scores terms based on how frequently they appear in the result set and how rarely they appear in the entire corpus. The `significantTerms` function emits a tuple for each term which contains the term, the score, the foreground count and the background count. The foreground count is how many documents the term appears in in the result set. The background count is how many documents the term appears in in the entire corpus. The foreground and background counts are global for the collection.
-
-=== significantTerms Parameters
-
-* `collection`: (Mandatory) The collection that the function is run on.
-* `q`: (Mandatory) The query that describes the foreground document set.
-* `field`: (Mandatory) The field to extract the terms from.
-* `limit`: (Optional, Default 20) The max number of terms to return.
-* `minDocFreq`: (Optional, Defaults to 5 documents) The minimum number of documents the term must appear in on a shard. This is a float value. If greater then 1.0 then it's considered the absolute number of documents. If less then 1.0 it's treated as a percentage of documents.
-* `maxDocFreq`: (Optional, Defaults to 30% of documents) The maximum number of documents the term can appear in on a shard. This is a float value. If greater then 1.0 then it's considered the absolute number of documents. If less then 1.0 it's treated as a percentage of documents.
-* `minTermLength`: (Optional, Default 4) The minimum length of the term to be considered significant.
-
-=== significantTerms Syntax
-
-[source,text]
-----
-significantTerms(collection1,
- q="body:Solr",
- field="author",
- limit="50",
- minDocFreq="10",
- maxDocFreq=".20",
- minTermLength="5")
-----
-
-In the example above the `significantTerms` function is querying `collection1` and returning at most 50 significant terms from the `authors` field that appear in 10 or more documents but not more then 20% of the corpus.
-
-== shortestPath
-
-The `shortestPath` function is an implementation of a shortest path graph traversal. The `shortestPath` function performs an iterative breadth-first search through an unweighted graph to find the shortest paths between two nodes in a graph. The `shortestPath` function emits a tuple for each path found. Each tuple emitted will contain a `path` key which points to a `List` of nodeIDs comprising the path.
-
-=== shortestPath Parameters
-
-* `collection`: (Mandatory) The collection that the topic query will be run on.
-* `from`: (Mandatory) The nodeID to start the search from
-* `to`: (Mandatory) The nodeID to end the search at
-* `edge`: (Mandatory) Syntax: `from_field=to_field`. The `from_field` defines which field to search from. The `to_field` defines which field to search to. See example below for a detailed explanation.
-* `threads`: (Optional : Default 6) The number of threads used to perform the partitioned join in the traversal.
-* `partitionSize`: (Optional : Default 250) The number of nodes in each partition of the join.
-* `fq`: (Optional) Filter query
-* `maxDepth`: (Mandatory) Limits to the search to a maximum depth in the graph.
-
-=== shortestPath Syntax
-
-[source,text]
-----
-shortestPath(collection,
- from="john@company.com",
- to="jane@company.com",
- edge="from_address=to_address",
- threads="6",
- partitionSize="300",
- fq="limiting query",
- maxDepth="4")
-----
-
-The expression above performs a breadth-first search to find the shortest paths in an unweighted, directed graph.
-
-The search starts from the nodeID "\john@company.com" in the `from_address` field and searches for the nodeID "\jane@company.com" in the `to_address` field. This search is performed iteratively until the `maxDepth` has been reached. Each level in the traversal is implemented as a parallel partitioned nested loop join across the entire collection. The `threads` parameter controls the number of threads performing the join at each level, while the `partitionSize` parameter controls the of number of nodes in each join partition. The `maxDepth` parameter controls the number of levels to traverse. `fq` is a limiting query applied to each level in the traversal.
-
-== shuffle
-
-The `shuffle` expression sorts and exports entire result sets. The `shuffle` expression is similar to the `search` expression except that
-under the covers `shuffle` always uses the /export handler. The `shuffle` expression is designed to be combined with the relational algebra
-decorators that require complete, sorted result sets. Shuffled result sets can be partitioned across worker nodes with the parallel
-stream decorator to perform parallel relational algebra. When used in parallel mode the partitionKeys parameter must be provided.
-
-=== shuffle Parameters
-
-* `collection`: (Mandatory) the collection being searched.
-* `q`: (Mandatory) The query to perform on the Solr index.
-* `fl`: (Mandatory) The list of fields to return.
-* `sort`: (Mandatory) The sort criteria.
-* `zkHost`: Only needs to be defined if the collection being searched is found in a different zkHost than the local stream handler.
-* `partitionKeys`: Comma delimited list of keys to partition the search results by. To be used with the parallel function for parallelizing operations across worker nodes. See the <<stream-decorators.adoc#parallel,parallel>> function for details.
-
-=== shuffle Syntax
-
-[source,text]
-----
-shuffle(collection1,
- q="*:*",
- fl="id,a_s,a_i,a_f",
- sort="a_f asc, a_i asc")
-----
-
-== stats
-
-The `stats` function gathers simple aggregations for a search result set. The stats function does not support rollups over buckets, so the stats stream always returns a single tuple with the rolled up stats. Under the covers the stats function pushes down the generation of the stats into the search engine using the StatsComponent. The stats function currently supports the following metrics: `count(*)`, `sum()`, `avg()`, `min()`, and `max()`.
-
-=== stats Parameters
-
-* `collection`: (Mandatory) Collection the stats will be aggregated from.
-* `q`: (Mandatory) The query to build the aggregations from.
-* `metrics`: (Mandatory) The metrics to include in the result tuple. Current supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)` and `count(*)`
-
-=== stats Syntax
-
-[source,text]
-----
-stats(collection1,
- q=*:*,
- sum(a_i),
- sum(a_f),
- min(a_i),
- min(a_f),
- max(a_i),
- max(a_f),
- avg(a_i),
- avg(a_f),
- count(*))
-----
-
-== timeseries
-
-The `timeseries` function builds a time series aggregation. Under the covers the `timeseries` function uses the
-JSON Facet API as its high performance aggregation engine.
-
-=== timeseries Parameters
-
-* `collection`: (Mandatory) Collection the stats will be aggregated from.
-* `q`: (Mandatory) The query to build the aggregations from.
-* `field`: (Mandatory) The date field for the time series.
-* `start`: (Mandatory) The start of the time series expressed in Solr date or date math syntax.
-* `end`: (Mandatory) The end of the time series expressed in Solr date or date math syntax.
-* `gap`: (Mandatory) The time gap between time series aggregation points expressed in Solr date math syntax.
-* `format`: (Optional) Date template to format the date field in the output tuples. Formatting is performed by Java's SimpleDateFormat class.
-* `metrics`: (Mandatory) The metrics to include in the result tuple. Current supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)` and `count(*)`
-
-=== timeseries Syntax
-
-[source,text]
-----
-timeseries(collection1,
- q=*:*,
- field="rec_dt"
- start="NOW-30DAYS",
- end="NOW",
- gap="+1DAY",
- format="YYYY-MM-dd",
- sum(a_i),
- max(a_i),
- max(a_f),
- avg(a_i),
- avg(a_f),
- count(*))
-----
-
-== train
-
-The `train` function trains a Logistic Regression text classifier on a training set stored in a SolrCloud collection. It uses a parallel iterative, batch Gradient Descent approach to train the model. The training algorithm is embedded inside Solr so with each iteration only the model is streamed across the network.
-
-The `train` function wraps a <<features,features>> function which provides the terms and inverse document frequency (IDF) used to train the model. The `train` function operates over the same training set as the `features` function, which includes both positive and negative examples of the class.
-
-With each iteration the `train` function emits a tuple with the model. The model contains the feature terms, weights, and the confusion matrix for the model. The optimized model can then be used to classify documents based on their feature terms.
-
-=== train Parameters
-
-* `collection`: (Mandatory) Collection that holds the training set
-* `q`: (Mandatory) The query that defines the training set. The IDF for the features will be generated on the
-* `name`: (Mandatory) The name of model. This can be used to retrieve the model if they stored in a Solr Cloud collection.
-* `field`: (Mandatory) The text field to extract the features from.
-* `outcome`: (Mandatory) The field that defines the class, positive or negative
-* `maxIterations`: (Mandatory) How many training iterations to perform.
-* `positiveLabel`: (defaults to 1) The value in the outcome field that defines a positive outcome.
-
-=== train Syntax
-
-[source,text]
-----
-train(collection1,
- features(collection1, q="*:*", featureSet="first", field="body", outcome="out_i", numTerms=250),
- q="*:*",
- name="model1",
- field="body",
- outcome="out_i",
- maxIterations=100)
-----
-
-== topic
-
-The `topic` function provides publish/subscribe messaging capabilities built on top of SolrCloud. The topic function allows users to subscribe to a query. The function then provides one-time delivery of new or updated documents that match the topic query. The initial call to the topic function establishes the checkpoints for the specific topic ID. Subsequent calls to the same topic ID will return documents added or updated after the initial checkpoint. Each run of the topic query updates the checkpoints for the topic ID. Setting the initialCheckpoint parameter to 0 will cause the topic to process all documents in the index that match the topic query.
-
-[WARNING]
-====
-The topic function should be considered in beta until https://issues.apache.org/jira/browse/SOLR-8709[SOLR-8709] is committed and released.
-====
-
-=== topic Parameters
-
-* `checkpointCollection`: (Mandatory) The collection where the topic checkpoints are stored.
-* `collection`: (Mandatory) The collection that the topic query will be run on.
-* `id`: (Mandatory) The unique ID for the topic. The checkpoints will be saved under this id.
-* `q`: (Mandatory) The topic query.
-* `fl`: (Mandatory) The field list returned by the topic function.
-* `initialCheckpoint`: (Optional) Sets the initial Solr `\_version_` number to start reading from the queue. If not set, it defaults to the highest version in the index. Setting to 0 will process all records that match query in the index.
-
-=== topic Syntax
-
-[source,text]
-----
-topic(checkpointCollection,
- collection,
- id="uniqueId",
- q="topic query",
- fl="id, name, country")
-----
-
-== tuple
-
-The `tuple` function emits a single Tuple with name/value pairs. The values can be set to variables assigned in a `let` expression, literals, Stream Evaluators or
-Stream Expressions. In the case of Stream Evaluators the tuple will output the return value from the evaluator.
-This could be a numeric, list or map. If a value is set to a Stream Expression, the `tuple` function will flatten
-the tuple stream from the Stream Expression into a list of Tuples.
-
-=== tuple Parameters
-
-* name=value pairs
-
-=== tuple Syntax
-
-[source,text]
-----
-tuple(a=add(1,1),
- b=search(collection1, q="cat:a", fl="a, b, c", sort="a desc"))
-----
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9beafb61/solr/solr-ref-guide/src/streaming-expressions.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/streaming-expressions.adoc b/solr/solr-ref-guide/src/streaming-expressions.adoc
index 6d13573..95402ef 100644
--- a/solr/solr-ref-guide/src/streaming-expressions.adoc
+++ b/solr/solr-ref-guide/src/streaming-expressions.adoc
@@ -1,7 +1,7 @@
= Streaming Expressions
:page-shortname: streaming-expressions
:page-permalink: streaming-expressions.html
-:page-children: stream-sources, stream-decorators, stream-evaluators, statistical-programming, graph-traversal
+:page-children: stream-source-reference, stream-decorator-reference, stream-evaluator-reference, statistical-programming, graph-traversal
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
@@ -121,12 +121,12 @@ Because streaming expressions relies on the `/export` handler, many of the field
Stream sources originate streams. The most commonly used one of these is `search`, which does a query.
-A full reference to all available source expressions is available in <<stream-sources.adoc#stream-sources,Stream Sources>>.
+A full reference to all available source expressions is available in <<stream-source-reference.adoc#stream-source-reference,Stream Source Reference>>.
=== About Stream Decorators
Stream decorators wrap other stream functions or perform operations on a stream.
-A full reference to all available decorator expressions is available in <<stream-decorators.adoc#stream-decorators,Stream Decorators>>.
+A full reference to all available decorator expressions is available in <<stream-decorator-reference.adoc#stream-decorator-reference,Stream Decorator Reference>>.
=== About Stream Evaluators
@@ -141,4 +141,4 @@ In cases where you want to use raw values as part of an evaluation you will need
If you wish to use a raw string as part of an evaluation, you will want to consider using the `raw(string)` evaluator. This will always return the raw value, no matter what is entered.
-A full reference to all available evaluator expressions is available in <<stream-evaluators.adoc#stream-evaluators,Stream Evaluators>>.
+A full reference to all available evaluator expressions is available in <<stream-evaluator-reference.adoc#stream-evaluator-reference,Stream Evaluator Reference>>.