You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by "Hoss Man (Confluence)" <co...@apache.org> on 2013/11/15 00:08:00 UTC
[CONF] Apache Solr Reference Guide > Function Queries
Space: Apache Solr Reference Guide (https://cwiki.apache.org/confluence/display/solr)
Page: Function Queries (https://cwiki.apache.org/confluence/display/solr/Function+Queries)
Change Comment:
---------------------------------------------------------------------
tweak formating and link to [Working with Dates]
Edited by Hoss Man:
---------------------------------------------------------------------
{section}
{column:width=75%}
Function queries enable you to generate a relevancy score using the actual value of one or more numeric fields. Function queries are supported by the [DisMax|The DisMax Query Parser], [Extended DisMax|The Extended DisMax Query Parser], and [standard|The Standard Query Parser] query parsers.
Function queries use _functions_. The functions can be a constant (numeric or string literal), a field, another function or a parameter substitution argument. You can use these functions to modify the ranking of results for users. These could be used to change the ranking of results based on a user's location, or some other calculation.
{column}
{column:width=25%}
{panel}
Function query topics covered in this section:
{toc}
{panel}
{column}
{section}
h2. Using Function Query
Functions must be expressed as function calls (for example, {{sum(a,b)}} instead of simply {{a+b}}).
There are three principal ways of including function queries in a Solr query:
* Introduce a function query with the {{\_val\_}} keyword. For example:
\\
{code:borderStyle=solid|borderColor=#666666}
_val_:mynumericfield _val_:"recip(rord(myfield),1,2,3)"
{code}
* Use a parameter that has an explicit type of FunctionQuery, such as the DisMax query parser's [{{bf}} (boost function) parameter|The DisMax Query Parser#The {{bf}} (Boost Functions) Parameter]. Note that the {{bf}} parameter actually takes a list of function queries separated by white space and each with an optional boost. Make sure you eliminate any internal white space in single function queries when using {{bf}}. For example:
\\
{code:borderStyle=solid|borderColor=#666666}
q=dismax&bf="ord(popularity)^0.5 recip(rord(price),1,1000,1000)^0.3"
{code}
* Add the results of FunctionQueries as fields to a document. For instance, for:
\\
{code:borderStyle=solid|borderColor=#666666}
&fl=sum(x, y),id,a,b,c,score
{code}
the output would be:
\\
{code:borderStyle=solid|borderColor=#666666}
...
<str name="id">foo</str>
<float name="sum(x,y)">40</float>
<float name="score">0.343</float>
...
{code}
Only functions with fast random access are recommended.
{topofpage}
h2. Available Functions
The table below summarizes the functions available for function queries.
|| Function || Description || Syntax Examples ||
| abs | Returns the absolute value of the specified value or function. | {{abs\(x)}} \\ {{abs(-5)}} |
| and | Returns a value of true if and only if both of its operands are true. | {{and("blue","sky")}} is true only if the sky is blue. |
| constant | Specifies a floating point constant. | {{1.5}} \\ {{\_val_:1.5}} |
| def | {{def}} is short for default. Returns the value of field "field", or if the field does not exist, returns the default value specified. and yields the first value where {{exists()==true}}.) | {{def(rating,5):}} This {{def()}} function returns the rating, or if no rating specified in the doc, returns 5 \\
{{def(myfield, 1.0):}} equivalent to {{if(exists(myfield),myfield,1.0)}} |
| div | Divides one value or function by another. div(x,y) divides x by y. | {{div(1,y)}} \\ {{div(sum(x,100),max(y,1))}} |
| dist | Return the distance between two vectors (points) in an n-dimensional space. Takes in the power, plus two or more ValueSource instances and calculates the distances between the two vectors. Each ValueSource must be a number. There must be an even number of ValueSource instances passed in and the method assumes that the first half represent the first vector and the second half represent the second vector. | {{dist(2, x, y, 0, 0):}} calculates the Euclidean distance between (0,0) and (x,y) for each document \\ \\ {{dist(1, x, y, 0, 0)}}: calculates the Manhattan (taxicab) distance between (0,0) and (x,y) for each document \\ \\ {{dist(2, x,y,z,0,0,0):}} Euclidean distance between (0,0,0) and (x,y,z) for each document. \\ \\ {{dist(1,x,y,z,e,f,g)}}: Euclidean distance between (x,y,z) and (e,f,g) where each letter is a field name |
| docfreq(field,val) | Returns the number of documents that contain the term in the field. This is a constant (the same value for all documents in the index). \\
\\
You can quote the term if it's more complex, or do parameter substitution for the term value. | {{docfreq(text,solr)}} \\
\\
{{docfreq(text,'solr')}} \\
\\
&{{defType=func&q=docfreq(text,$myterm)&myterm=solr}} |
| exists | Returns TRUE if any member of the field exists. | {{exists(author)}} returns TRUE for any document has a value in the "author" field.\\
\\
{{exists(query(price:5.00))}} returns TRUE if "price" contains "5.00". |
| field | Returns the numeric field value of an indexed (not multi-valued) field with a maximum of one value per document. The syntax is simply the field name by itself. 0 is returned for documents without a value in the field. | {{\_val_:myFloatField}} |
| hsin | The Haversine distance calculates the distance between two points on a sphere when traveling along the sphere. The values must be in radians. {{hsin}} also take a Boolean argument to specify whether the function should convert its output to radians. | {{hsin(2, true, x, y, 0, 0)}} |
| fl | Defines the fields of a document that are returned in response to a query. You need to specify {{score}} as one of the fields to return if you want any calculated values to be returned in the score. \\ | {{fl=*,score}} \\ |
| idf | Inverse document frequency; a measure of whether the term is common or rare across all documents. Obtained by dividing the total number of documents by the number of documents containing the term, and then taking the logarithm of that quotient. See also {{tf}}. \\ | {{idf(list1,326.55)}}: measures the inverse of the frequency of the occurrence of the number 326.55 in {{list1}}. |
| if | Enables conditional function queries. {{IF(test; value1; value2)}} where: \\
* {{test}} is or refers to a logical value or expression that returns a logical value (TRUE or FALSE).
* {{value1}} is the value that is returned by the function if test yields TRUE.
* {{value2}} is the value that is returned by the function if test yields FALSE.
* If {{value2}} is omitted it is assumed to be FALSE; if {{value1}} is also omitted it is assumed to be TRUE. \\
An expression can be any function which outputs boolean values, or even functions returning numeric values, in which case value 0 will be interpreted as false, or strings, in which case empty string is interpreted as false. | {{if(color=="red"; 100; if(color=="green"; 50; 25))}} : \\
This function checks the document field "color", and if it is "red" returns 100, if it is "green" returns 50, else returns 25. |
| linear | Implements {{m*x+c}} where {{m}} and {{c}} are constants and {{x}} is an arbitrary function. This is equivalent to {{sum(product(m,x),c)}}, but slightly more efficient as it is implemented as a single function. | {{linear(x,m,c)}} \\ {{linear(x,2,4)}} returns {{2*x+4}} |
| log | Returns the log base 10 of the specified function. | {{log(x)log(sum(x,100))}} |
| map | Maps any values of the function x that fall within min and max inclusive to the specified target. The arguments min,max,target are constants. The function outputs the field's value if it does not fall between min and max. | {{map(x,min,max,target)}} \\ {{map(x,0,0,1)}} \- changes any values of 0 to 1. This can be useful in handling default 0 values. \\ {{map(x,min,max,target,altarg)}} \\ \\ {{map(x,0,0,1,0)}} \- changes any values of 0 to 1 and if the value is not zero it can be set to the value of the 5th argument instead of defaulting to the field's value. |
| max | Returns the max of another function and a constant, which are specified as arguments: {{max(x,c)}}. The max function is useful for "bottoming out" another function at some constant. | {{max(myfield,0)}} |
| maxdoc | Returns the number of documents in the index, including those that are marked as deleted but have not yet been purged. This is a constant (the same value for all documents in the index). | {{maxdoc(list1)}} |
| ms | Returns milliseconds of difference between its arguments. Dates are relative to the Unix or POSIX time epoch, midnight, January 1, 1970 UTC. Arguments may be the name of an indexed {{TrieDateField}}, or date math based on a [constant date or {{NOW}}|Working with Dates]. \\
{{ms()}}: Equivalent to {{ms(NOW)}}, number of milliseconds since the epoch. \\ {{ms(a):}} Returns the number of milliseconds since the epoch that the argument represents. \\ {{ms(a,b)}} : Returns the number of milliseconds that b occurs before a (that is, a - b) \\ | {{ms(NOW/DAY)}} \\ {{ms(2000-01-01T00:00:00Z)}} \\ {{ms(mydatefield)}} \\ {{ms(NOW,mydatefield)}} \\ {{ms(mydatefield,2000-01-01T00:00:00Z)}} \\ {{ms(datefield1,datefield2)}} |
| norm(_field_) | Returns the "norm" stored in the index, the product of the index time boost and the length normalization factor, according to the [Similarity|http://lucene.apache.org/java/3_0_0/api/core/org/apache/lucene/search/Similarity.html%7CSimilarity] for the field. | {{norm(text)}} |
| not | A logically negative value. | {{if (NOT value1) \[ ... \]}}: TRUE only when {{value1}} is false. |
| numdocs | Returns the number of documents in the index, not including those that are marked as deleted but have not yet been purged. This is a constant (the same value for all documents in the index). | {{numdocs(list1)}} |
| or | A logical disjunction. \\ | {{(value1 OR value2):}} TRUE if either {{value1}} or {{value2}} is true. \\ |
| ord | Returns the ordinal of the indexed field value within the indexed list of terms for that field in Lucene index order (lexicographically ordered by unicode value), starting at 1. In other words, for a given field, all values are ordered lexicographically; this function then returns the offset of a particular value in that ordering. The field must have a maximum of one value per document (not multi-valued). 0 is returned for documents without a value in the field. \\ \\
{note}{{{}ord()}} depends on the position in an index and can change when other documents are inserted or deleted.{note}\\ \\
See also {{rord}} below. | {{ord(myIndexedField)}} \\ {{\_val_:"ord(myIndexedField)"}} \\
\\
Example: If there were only three values ("apple","banana","pear") for a particular field, then: {{ord("apple")=1ord("banana")=2ord("pear")=3}} |
| pow | Raises the specified base to the specified power. {{pow(x,y)}} raises x to the power of y. | {{pow(x,y)}} \\
{{pow(x,log\(y))}} \\
{{pow(x,0.5):}} the same as {{sqrt}} |
| product | Returns the product of multiple values or functions, which are specified in a comma-separated list. | {{product(x,y,...)}} \\
{{product(x,2)}} \\
{{product(x,y)}} |
| query | Returns the score for the given subquery, or the default value for documents not matching the query. Any type of subquery is supported through either parameter de-referencing {{$otherparam}} or direct specification of the query string in the [Local Parameters|solr:Local Parameters in Queries] through the {{v}} key. | {{query(subquery, default)}} \\ \\
{{q=product(popularity, query(\{\!dismax v='solr rocks'})}}: returns the product of the popularity and the score of the DisMax query. \\ \\
{{q=product(popularity, query($qq))&qq=\{\!dismax}solr rocks}}: equivalent to the previous query, using parameter de-referencing. \\ \\
{{q=product(popularity, query($qq,0.1))&qq=\{\!dismax}solr rocks}}: specifies a default score of 0.1 for documents that don't match the DisMax query. |
| recip | Performs a reciprocal function with {{recip(myfield,m,a,b)}} implementing {{a/(m*x+b)}}. {{m,a,b}} are constants, and {{x}} is any arbitrarily complex function. \\
When a and b are equal, and x>=0, this function has a maximum value of 1 that drops as x increases. Increasing the value of a and b together results in a movement of the entire function to a flatter part of the curve. These properties can make this an ideal function for boosting more recent documents when x is {{rord(datefield)}}. | {{recip(myfield,m,a,b)}} \\ {{recip(rord(creationDate),1,1000,1000)}} |
| rord | Returns the reverse ordering of that returned by {{ord}}. | {{rord(myDateField)}} \\ {{\_val_:"rord(myDateField)"}} \\ \\ {{rord(myDateField)}}: a metric for how old a document is. The youngest document will return 1. The oldest document will return the total number of documents. |
| scale | Scales values of the function x such that they fall between the specified {{minTarget}} and {{maxTarget}} inclusive. The current implementation traverses all of the function values to obtain the min and max, so it can pick the correct scale. \\ \\
The current implementation cannot distinguish when documents have been deleted or documents that have no value. It uses 0.0 values for these cases. This means that if values are normally all greater than 0.0, one can still end up with 0.0 as the min value to map from. In these cases, an appropriate map() function could be used as a workaround to change 0.0 to a value in the real range, as shown here: \\
scale(map(x,0,0,5),1,2) | {{scale(x,minTarget,maxTarget)}} \\ {{scale(x,1,2)}}: scales the values of x such that all values will be between 1 and 2 inclusive. |
| sqedist | The Square Euclidean distance calculates the 2-norm (Euclidean distance) but does not take the square root, thus saving a fairly expensive operation. It is often the case that applications that care about Euclidean distance do not need the actual distance, but instead can use the square of the distance. There must be an even number of ValueSource instances passed in and the method assumes that the first half represent the first vector and the second half represent the second vector. | {{sqedist(x_td, y_td, 0, 0)}} |
| sqrt | Returns the square root of the specified value or function. | {{sqrt(x)sqrt(100)sqrt(sum(x,100))}} |
| strdist | Calculate the distance between two strings. Uses the Lucene spell checker {{StringDistance}} interface and supports all of the implementations available in that package, plus allows applications to plug in their own via Solr's resource loading capabilities. {{strdist}} takes (string1, string2, distance measure). Possible values for distance measure are: \\ \\
jw: Jaro-Winkler \\ \\
edit: Levenstein or Edit distance \\ \\
ngram: The NGramDistance, if specified, can optionally pass in the ngram size too. Default is 2. \\ \\
FQN: Fully Qualified class Name for an implementation of the StringDistance interface. Must have a no-arg constructor. | {{strdist("SOLR",id,edit)}} |
| sub | Returns x-y from sub(x,y). | {{sub(myfield,myfield2)}} \\ {{sub(100,sqrt(myfield))}} |
| sum | Returns the sum of multiple values or functions, which are specified in a comma-separated list. | {{sum(x,y,...) sum(x,1)}} \\ {{sum(x,y)}} \\ {{sum(sqrt\(x),log\(y),z,0.5)}} |
| sumtotaltermfreq | Returns the sum of {{totaltermfreq}} values for all terms in the field in the entire index (i.e., the number of indexed tokens for that field). (Aliases {{sumtotaltermfreq}} to {{sttf}}.) | If doc1:(fieldX:A B C) and doc2:(fieldX:A A A A): \\
{{docFreq(fieldX:A)}} = 2 (A appears in 2 docs) \\
{{freq(doc1, fieldX:A)}} = 4 (A appears 4 times in doc 2) \\
{{totalTermFreq(fieldX:A)}} = 5 (A appears 5 times across all docs) \\
{{sumTotalTermFreq(fieldX)}} = 7 in {{fieldX}}, there are 5 As, 1 B, 1 C |
| termfreq | Returns the number of times the term appears in the field for that document. | {{termfreq(text,'memory')}} |
| tf | Term frequency; returns the term frequency factor for the given term, using the [Similarity|http://lucene.apache.org/java/3_0_0/api/core/org/apache/lucene/search/Similarity.html] for the field. The {{tf-idf}} value increases proportionally to the number of times a word appears in the document, but is offset by the frequency of the word in the document, which helps to control for the fact that some words are generally more common than others. See also {{idf}}. \\ | {{tf(text,'solr')}} |
| top | Causes the function query argument to derive its values from the top-level IndexReader containing all parts of an index. For example, the ordinal of a value in a single segment will be different from the ordinal of that same value in the complete index. \\
The {{ord()}} and {{rord()}} functions implicitly use {{top()}}, and hence {{ord(foo)}} is equivalent to {{top(ord(foo))}}. | |
| Totaltermfreq | Returns the number of times the term appears in the field in the entire index. (Aliases {{totaltermfreq}} to {{ttf}}.) | {{ttf(text,'memory')}} |
| xor() | Logical exclusive disjunction, or one or the other but not both. | {{xor(field1,field2)}} returns TRUE if either {{field1}} or {{field2}} is true; FALSE if both are true. |
{topofpage}
h2. Example Function Queries
To give you a better understanding of how function queries can be used in Solr, suppose an index stores the dimensions in meters x,y,z of some hypothetical boxes with arbitrary names stored in field {{boxname}}. Suppose we want to search for box matching name {{findbox}} but ranked according to volumes of boxes. The query parameters would be:
{{q=boxname:findbox_val_:"product(product(x,y),z)}}
This query will rank the results based on volumes. In order to get the computed volume, you will need to request the {{score}}, which will contain the resultant volume:
{{&fl=*, score}}
Suppose that you also have a field storing the weight of the box as {{weight}}. To sort by the density of the box and return the value of the density in score, you would submit the following query:
{code:borderStyle=solid|borderColor=#666666}
http://localhost:8983/solr/select/?q=boxname:findbox_val_div(weight,product(product(x,y),z))"&fl=boxname x y z weight score
{code}
{topofpage}
h2. Sort By Function
You can sort your query results by the output of a function. For example, to sort results by distance, you could enter:
{code:borderStyle=solid|borderColor=#666666}
http://localhost:8983/solr/select?q=*:*&sort=dist(2, point1, point2) desc
{code}
Sort by function also supports pseudo-fields: fields can be generated dynamically and return results as though it was normal field in the index. For example,
{{&fl=id,sum(x, y),score}}
would return:
{code:borderStyle=solid|borderColor=#666666}
<str name="id">foo</str>
<float name="sum(x,y)">40</float>
<float name="score">0.343</float>
{code}
{topofpage}
h2. Related Topics
* [FunctionQuery|https://wiki.apache.org/solr/FunctionQuery]
{scrollbar}
Stop watching space: https://cwiki.apache.org/confluence/users/removespacenotification.action?spaceKey=solr
Change email notification preferences: https://cwiki.apache.org/confluence/users/editmyemailsettings.action