You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@solr.apache.org by ep...@apache.org on 2023/12/05 20:15:17 UTC
(solr) branch branch_9x updated: SOLR-17016: Migrate tutorial content on ParamSets and Vectors into Ref Guide (#1988)
This is an automated email from the ASF dual-hosted git repository.
epugh pushed a commit to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/solr.git
The following commit(s) were added to refs/heads/branch_9x by this push:
new 0115d44831d SOLR-17016: Migrate tutorial content on ParamSets and Vectors into Ref Guide (#1988)
0115d44831d is described below
commit 0115d44831d81d38de8622c708e59263ef925501
Author: Eric Pugh <ep...@opensourceconnections.com>
AuthorDate: Tue Dec 5 15:14:32 2023 -0500
SOLR-17016: Migrate tutorial content on ParamSets and Vectors into Ref Guide (#1988)
Moved content out of the films readme and introduce new tutorials to the Ref Guide.
---------
Co-authored-by: Christine Poerschke <cp...@apache.org>
---
solr/example/films/README.md | 183 -------------------
.../getting-started/getting-started-nav.adoc | 2 +
.../getting-started/pages/tutorial-films.adoc | 2 +-
.../getting-started/pages/tutorial-paramsets.adoc | 170 +++++++++++++++++
.../getting-started/pages/tutorial-vectors.adoc | 203 +++++++++++++++++++++
5 files changed, 376 insertions(+), 184 deletions(-)
diff --git a/solr/example/films/README.md b/solr/example/films/README.md
index a2c5f8f3a01..0e7229df720 100644
--- a/solr/example/films/README.md
+++ b/solr/example/films/README.md
@@ -16,186 +16,3 @@ This data consists of the following fields:
are created by the use of an Update Request Processor Chain called `add-unknown-fields-to-the-schema`.
The `film_vector` is an embedding vector created to represent the movie with 10 dimensions. The vector is created from a BERT pre-trained model, followed by a dimension reduction technique to reduce the embeddings from 768 to 10 dimensions. Even though it is expected that similar movies will be close to each other, this model is just a "toy example", so it's not guaranteed to be a good representation for the movies. The Python scripts utilized to create the model and calculate the films [...]
-
- The below steps walk you through learning how to start up Solr, setup the films collection yourself, and then load data. We'll then create ParamSets to organize our query parameters. Finally, we also show some example of KNN queries exploring the Dense Vectors feature.
-
- You can also run `bin/solr start -example films` or `bin/solr start -c -example films` for SolrCloud version which does all the below steps for you.
-
- Steps:
- * Start Solr:
- ```
- bin/solr start
- ```
-
- * Create a "films" core:
-
- ```
- bin/solr create -c films
- ```
-
- * Set the schema on a couple of fields that Solr would otherwise guess differently (than we'd like) about:
-
- ```
- curl http://localhost:8983/solr/films/schema -X POST -H 'Content-type:application/json' --data-binary '{
- "add-field-type" : {
- "name":"knn_vector_10",
- "class":"solr.DenseVectorField",
- "vectorDimension":10,
- "similarityFunction":"cosine",
- "knnAlgorithm":"hnsw"
- },
- "add-field" : [
- {
- "name":"name",
- "type":"text_general",
- "multiValued":false,
- "stored":true
- },
- {
- "name":"initial_release_date",
- "type":"pdate",
- "stored":true
- },
- {
- "name":"film_vector",
- "type":"knn_vector_10",
- "indexed":true,
- "stored":true
- }
- ]
- }'
- ```
-
- * Now let's index the data, using one of these three commands:
-
- - JSON: `bin/post -c films example/films/films.json`
- - XML: `bin/post -c films example/films/films.xml`
- - CSV:
- ```
- bin/post \
- -c films \
- example/films/films.csv \
- -params "f.genre.split=true&f.directed_by.split=true&f.film_vector.split=true&f.genre.separator=|&f.directed_by.separator=|&f.film_vector.separator=|"
- ```
-
-
- * Let's get searching!
- - Search for 'Batman':
-
- http://localhost:8983/solr/films/query?q=name:batman
-
- * If you get an error about the name field not existing, you haven't yet indexed the data
- * If you don't get an error, but zero results, chances are that the _name_ field schema type override wasn't set
- before indexing the data the first time (it ended up as a "string" type, requiring exact matching by case even).
- It's easiest to simply reset the environment and try again, ensuring that each step successfully executes.
-
- - Show me all 'Super hero' movies:
-
- http://localhost:8983/solr/films/query?q=*:*&fq=genre:%22Superhero%20movie%22
-
- - Let's see the distribution of genres across all the movies. See the facet section of the response for the counts:
-
- http://localhost:8983/solr/films/query?q=*:*&facet=true&facet.field=genre
-
- * Time for relevancy tuning with ParamSets :
-
- - Search for 'harry potter':
-
- http://localhost:8983/solr/films/query?q=name:harry%20potter
-
- * Notice the very first result is the move _Dumb & Dumberer: When Harry Met Lloyd_?
- That is clearly not what we are looking for.
-
- - Let's set up two relevancy algorithms, and then compare the quality of the results.
- Algorithm *A* will use the `dismax` and a `qf` parameters, and then Algorithm *B*
- will use `dismax`, `qf` and a must match `mm` of 100%.
-
- ```
- curl http://localhost:8983/solr/films/config/params -X POST -H 'Content-type:application/json' --data-binary '{
- "set": {
- "algo_a":{
- "defType":"dismax",
- "qf":"name"
- }
- },
- "set": {
- "algo_b":{
- "defType":"dismax",
- "qf":"name",
- "mm":"100%"
- }
- }
- }'
- ```
-
- - Search for 'harry potter' with Algorithm *A*:
-
- http://localhost:8983/solr/films/query?q=harry%20potter&useParams=algo_a
-
- * Now we are returning the five results, including the Harry Potter movies, however notice that we still have the
- _Dumb & Dumberer: When Harry Met Lloyd_ movie coming back?
-
- - Search for 'harry potter' with Algorithm *B*:
-
- http://localhost:8983/solr/films/query?q=harry%20potter&useParams=algo_b
-
- * We are now returning only the four Harry Potter movies, leading to more precise results!
- We can say that we believe Algorithm *B* is better then Algorithm *A*. You can extend
- this to online A/B testing very easily to confirm with real users.
-
- * Now let's search with Dense Vectors and KNN!
-
- - Before making the queries, we define an example target vector, simulating a person that
- watched 3 movies: _Finding Nemo_, _Bee Movie_, and _Harry Potter and the Chamber of Secrets_.
- We get the 3 vectors of each movie, then calculate the resulting average vector, which will
- be used as input vector for all the following example queries.
-
- ```
- [-0.1784, 0.0096, -0.1455, 0.4167, -0.1148, -0.0053, -0.0651, -0.0415, 0.0859, -0.1789]
- ```
- - Search for the top 10 movies most similar to the target vector (KNN Query for recommendation):
-
- http://localhost:8983/solr/films/query?q={!knn%20f=film_vector%20topK=10}[-0.1784,0.0096,-0.1455,0.4167,-0.1148,-0.0053,-0.0651,-0.0415,0.0859,-0.1789]
-
- * Notice that among the results, there are some animation family movies, such as _Curious George_ and _Bambi_, which makes sense, since the target vector was created with two other animation family movies (_Finding Nemo_ and _Bee Movie_).
- * We also notice that among the results there are two movies that the person already watched. In the next example we will filter then out.
-
- - Search for the top 10 movies most similar to the resulting vector, excluding the movies already watched (KNN query with Filter Query):
-
- http://localhost:8983/solr/films/query?q={!knn%20f=film_vector%20topK=10}[-0.1784,0.0096,-0.1455,0.4167,-0.1148,-0.0053,-0.0651,-0.0415,0.0859,-0.1789]&fq=-id:("%2Fen%2Ffinding_nemo"%20"%2Fen%2Fbee_movie"%20"%2Fen%2Fharry_potter_and_the_chamber_of_secrets_2002")
-
- - Search for movies with "cinderella" in the name among the top 50 movies most similar to the target vector (KNN as Filter Query):
-
- http://localhost:8983/solr/films/query?q=name:cinderella&fq={!knn%20f=film_vector%20topK=50}[-0.1784,0.0096,-0.1455,0.4167,-0.1148,-0.0053,-0.0651,-0.0415,0.0859,-0.1789]
-
- * There are 3 "cinderella" movies in the index, but only 1 is among the top 50 most similar to the target vector (_Cinderella III: A Twist in Time_).
-
- - Search for movies with "animation" in the genre, and rerank the top 5 documents by combining (sum) the original query score with twice (2x) the similarity to the target vector (KNN with ReRanking):
-
- http://localhost:8983/solr/films/query?q=genre:animation&rqq={!knn%20f=film_vector%20topK=10000}[-0.1784,0.0096,-0.1455,0.4167,-0.1148,-0.0053,-0.0651,-0.0415,0.0859,-0.1789]&rq={!rerank%20reRankQuery=$rqq%20reRankDocs=5%20reRankWeight=2}
-
- * To guarantee we calculate the vector similarity score for all the movies, we set `topK=10000`, a number higher than the total number of documents (`1100`).
-
- * It's possible to combine the vector similarity scores with other scores, by using Sub-query,
- Function Queries and Parameter Dereferencing Solr features:
-
- - Search for "harry potter" movies, ranking the results by the similarity to the target vector instead of the lexical query score. Beside the `q` parameter, we define a "sub-query" named `q_vector`, that will calculate the similarity score between all the movies (since we set `topK=10000`). Then we use the sub-query parameter name as input for the `sort`, specifying that we want to rank descending according to the vector similarity score (`sort=$q_vector desc`):
-
- http://localhost:8983/solr/films/query?q=name:"harry%20potter"&q_vector={!knn%20f=film_vector%20topK=10000}[-0.1784,0.0096,-0.1455,0.4167,-0.1148,-0.0053,-0.0651,-0.0415,0.0859,-0.1789]&sort=$q_vector%20desc
-
- - Search for movies with "the" in the name, keeping the original lexical query ranking, but returning only movies with similarity to the target vector of 0.8 or higher. Like previously, we define the sub-query `q_vector`, but this time we use it as input for the `frange` filter, specifying that we want documents with at least 0.8 of vector similarity score:
-
- http://localhost:8983/solr/films/query?q=name:the&q_vector={!knn%20f=film_vector%20topK=10000}[-0.1784,0.0096,-0.1455,0.4167,-0.1148,-0.0053,-0.0651,-0.0415,0.0859,-0.1789]&fq={!frange%20l=0.8}$q_vector
-
- - Search for "batman" movies, ranking the results by combining 70% of the original lexical query score and 30% of the similarity to the target vector. Besides the `q` main query and the `q_vector` sub-query, we also specify the `q_lexical` query, which will hold the lexical score of the main `q` query. Then we specify a parameter variable called `score_combined`, which scales the lexical and similarity scores, applies the 0.7 and 0.3 weights, then sum the result. We set the `sort` p [...]
-
- http://localhost:8983/solr/films/query?q=name:batman&q_lexical={!edismax%20v=$q}&q_vector={!knn%20f=film_vector%20topK=10000}[-0.1784,0.0096,-0.1455,0.4167,-0.1148,-0.0053,-0.0651,-0.0415,0.0859,-0.1789]&score_combined=sum(mul(scale($q_lexical,0,1),0.7),mul(scale($q_vector,0,1),0.3))&sort=$score_combined%20desc&fl=name,score,$q_lexical,$q_vector,$score_combined
-
-
-FAQ:
- Why override the schema of the _name_ and _initial_release_date_ fields?
-
- Without overriding those field types, the _name_ field would have been guessed as a multi-valued string field type
- and _initial_release_date_ would have been guessed as a multi-valued `pdate` type. It makes more sense with this
- particular data set domain to have the movie name be a single valued general full-text searchable field,
- and for the release date also to be single valued.
diff --git a/solr/solr-ref-guide/modules/getting-started/getting-started-nav.adoc b/solr/solr-ref-guide/modules/getting-started/getting-started-nav.adoc
index e384f820703..47bea772000 100644
--- a/solr/solr-ref-guide/modules/getting-started/getting-started-nav.adoc
+++ b/solr/solr-ref-guide/modules/getting-started/getting-started-nav.adoc
@@ -30,6 +30,8 @@
** xref:tutorial-techproducts.adoc[]
** xref:tutorial-films.adoc[]
** xref:tutorial-diy.adoc[]
+** xref:tutorial-paramsets.adoc[]
+** xref:tutorial-vectors.adoc[]
** xref:tutorial-solrcloud.adoc[]
** xref:tutorial-aws.adoc[]
diff --git a/solr/solr-ref-guide/modules/getting-started/pages/tutorial-films.adoc b/solr/solr-ref-guide/modules/getting-started/pages/tutorial-films.adoc
index 45d8b0f4b31..2a3becad0c3 100644
--- a/solr/solr-ref-guide/modules/getting-started/pages/tutorial-films.adoc
+++ b/solr/solr-ref-guide/modules/getting-started/pages/tutorial-films.adoc
@@ -238,7 +238,7 @@ $ bin/post -c films example/films/films.json
[,console]
----
-$ java -jar -Dc=films -Dauto example\exampledocs\post.jar example\films\*.json
+$ bin/solr post -c films example\films\films.json
----
====
--
diff --git a/solr/solr-ref-guide/modules/getting-started/pages/tutorial-paramsets.adoc b/solr/solr-ref-guide/modules/getting-started/pages/tutorial-paramsets.adoc
new file mode 100644
index 00000000000..a6d083660c9
--- /dev/null
+++ b/solr/solr-ref-guide/modules/getting-started/pages/tutorial-paramsets.adoc
@@ -0,0 +1,170 @@
+= Exercise 4: Using ParamSets
+:experimental:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+[[exercise-4]]
+== Exercise 4: Using ParamSets
+
+This exercise will teach you to use ParamSets to group a number of different query parameters into a labelled grouping that you can refer to in your queries.
+
+=== Getting Ready
+
+Make sure you have a running Solr, following the steps in xref:tutorial-films.adoc#restart-solr[].
+Then go ahead to the next section.
+
+=== Create a New Collection
+
+[,console]
+----
+$ bin/solr create -c films
+----
+
+Because we didn't specify a ConfigSet, we will end up using the `_default` ConfigSet.
+We'll specify the specific schema for a couple of fields that Solr would otherwise guess differently (than we'd like) about:
+
+[,console]
+----
+$ curl http://localhost:8983/solr/films/schema -X POST -H 'Content-type:application/json' --data-binary '{
+ "add-field" : [
+ {
+ "name":"name",
+ "type":"text_general",
+ "multiValued":false,
+ "stored":true
+ },
+ {
+ "name":"initial_release_date",
+ "type":"pdate",
+ "stored":true
+ }
+ ]
+}'
+----
+
+[NOTE]
+====
+Without explicitly defining those field types, the _name_ field would have been guessed as a multi-valued string field type
+and _initial_release_date_ would have been guessed as a multi-valued `pdate` type. It makes more sense with this
+particular data set domain to have the movie name be a single valued general full-text searchable field,
+and for the release date also to be single valued.
+====
+
+=== Index the Data
+
+Now that we have updated our Schema, we need to index the sample film data, or, if you already have indexed it, then re-index it to take advantage of the new field definitions we added.
+
+[.dynamic-tabs]
+--
+[example.tab-pane#unixindexjson]
+====
+[.tab-label]*Linux/Mac*
+
+[,console]
+----
+$ bin/solr post -c films example/films/films.json
+
+----
+====
+
+[example.tab-pane#winindexjson]
+====
+[.tab-label]*Windows*
+
+[,console]
+----
+$ bin/solr post -c films example\films\films.json
+----
+====
+--
+
+=== Let's get Searching!
+
+Search for 'Batman':
+
+http://localhost:8983/solr/films/query?q=name:batman
+
+* If you get an error about the name field not existing, you haven't yet indexed the data.
+* If you don't get an error, but zero results, chances are that the _name_ field schema type override wasn't set
+ before indexing the data the first time (it ended up as a "string" type, requiring exact matching by case even).
+ It's easiest to simply reset your environment and try again, ensuring that each step successfully executes.
+
+Show me all 'Super hero' movies:
+
+[,console]
+----
+$ curl 'http://localhost:8983/solr/films/query?q=*:*&fq=genre:"Superhero movie"'
+----
+
+Let's see the distribution of genres across all the movies. See the facet section of the response for the counts:
+
+[,console]
+----
+$ curl 'http://localhost:8983/solr/films/query?q=*:*&facet=true&facet.field=genre'
+----
+
+=== Time for relevancy tuning with ParamSets :
+
+Now that we can query our data, let's actually use the ParamSets to organize our parameters into two experiments.
+
+Search for 'harry potter':
+
+http://localhost:8983/solr/films/query?q=name:harry%20potter
+
+Notice the very first result is the movie _Dumb & Dumberer: When Harry Met Lloyd_?
+That is clearly not related to any Harry Potter movies.
+
+Let's set up two relevancy algorithms, using our APIs, and then compare the quality of the results.
+Algorithm *A* will specify using `dismax` and a `qf` parameter, while Algorithm *B* will use `dismax`, `qf` and a must match `mm` set to 100%.
+
+[,console]
+----
+curl http://localhost:8983/solr/films/config/params -X POST -H 'Content-type:application/json' --data-binary '{
+"set": {
+ "algo_a":{
+ "defType":"dismax",
+ "qf":"name"
+ }
+ },
+ "set": {
+ "algo_b":{
+ "defType":"dismax",
+ "qf":"name",
+ "mm":"100%"
+ }
+ }
+}'
+----
+
+Search for 'harry potter' with Algorithm *A*:
+
+http://localhost:8983/solr/films/query?q=harry%20potter&useParams=algo_a
+
+We are returning the five results, including the Harry Potter movies, however notice that we still have the _Dumb & Dumberer: When Harry Met Lloyd_ movie coming back?
+
+Search for 'harry potter' with Algorithm *B*:
+
+http://localhost:8983/solr/films/query?q=harry%20potter&useParams=algo_b
+
+We are returning only the four Harry Potter movies, leading to more precise results!
+We can say that we believe Algorithm *B* is better then Algorithm *A*, at least for this one query.
+You can validate this hypothesis with online A/B testing to confirm with real users that Algorithm *B* is better overall.
+
+
+=== Exercise 4 Wrap Up
+
+In this exercise, we used the Schema API to create the fields that we needed, and then learned how to organize our query parameters into named groups of parameters called ParamSets that we created using the Config API and subsequently referenced in queries.
diff --git a/solr/solr-ref-guide/modules/getting-started/pages/tutorial-vectors.adoc b/solr/solr-ref-guide/modules/getting-started/pages/tutorial-vectors.adoc
new file mode 100644
index 00000000000..42a3174658f
--- /dev/null
+++ b/solr/solr-ref-guide/modules/getting-started/pages/tutorial-vectors.adoc
@@ -0,0 +1,203 @@
+= Exercise 5: Using Vectors
+:experimental:
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+[[exercise-5]]
+== Exercise 5: Using Vectors in Solr
+
+This exercise will use the Films example that we looked at previously in Exercise 4.
+
+=== Getting Ready
+
+Make sure you have a running Solr, following the steps in xref:tutorial-films.adoc#restart-solr[].
+Then go ahead to the next section.
+
+=== Preparing for the Vector data
+
+[,console]
+----
+$ bin/solr create -c films
+----
+
+Because we didn't specify a ConfigSet when we created the collection, we will use the `_default` ConfigSet.
+
+First we need to update our schema to add the vector field type, the field to hold the vector values and some supporting fields.
+
+[,console]
+----
+$ curl http://localhost:8983/solr/films/schema -X POST -H 'Content-type:application/json' --data-binary '{
+ "add-field-type" : {
+ "name":"knn_vector_10",
+ "class":"solr.DenseVectorField",
+ "vectorDimension":10,
+ "similarityFunction":"cosine",
+ "knnAlgorithm":"hnsw"
+ },
+ "add-field" : [
+ {
+ "name":"film_vector",
+ "type":"knn_vector_10",
+ "indexed":true,
+ "stored":true
+ },
+ {
+ "name":"name",
+ "type":"text_general",
+ "multiValued":false,
+ "stored":true
+ },
+ {
+ "name":"initial_release_date",
+ "type":"pdate",
+ "stored":true
+ }
+ ]
+}'
+----
+
+=== Now index the Films data with Vectors
+
+We have the vectors embedded in our `films.json` file, so let's index that data, taking advantage of our new schema field we just defined.
+
+[.dynamic-tabs]
+--
+[example.tab-pane#unixindexjson]
+====
+[.tab-label]*Linux/Mac*
+
+[,console]
+----
+$ bin/solr post -c films example/films/films.json
+
+----
+====
+
+[example.tab-pane#winindexjson]
+====
+[.tab-label]*Windows*
+
+[,console]
+----
+$ bin/solr post -c films example\films\films.json
+----
+====
+--
+
+=== Let's do some Vector searches
+Before making the queries, we define an example target vector, simulating a person that
+watched 3 movies: _Finding Nemo_, _Bee Movie_, and _Harry Potter and the Chamber of Secrets_.
+We get the vector of each movie, then calculate the resulting average vector, which will
+be used as the input vector for all the following example queries.
+
+```
+[-0.1784, 0.0096, -0.1455, 0.4167, -0.1148, -0.0053, -0.0651, -0.0415, 0.0859, -0.1789]
+```
+
+[NOTE]
+====
+Interested in calculating the vector using Solr's xref:query-guide:streaming-expressions.adoc[streaming capability]?
+Here is an example of a streaming expression that you can run via the Solr Admin Stream UI:
+```
+let(
+ a=select(
+ search(films,
+ qt="/select",
+ q="name:"Finding Nemo" OR name:"Bee Movie" OR name:"Harry Potter and the Chamber of Secrets"",
+ fl="id,name,film_vector"),
+ film_vector),
+ b=col(a, film_vector),
+ m=matrix(valueAt(b, 0), valueAt(b, 1), valueAt(b, 2)),
+ average=scalarDivide(3, sumColumns(m))
+ )
+```
+
+The output is:
+```
+{
+ "result-set": {
+ "docs": [
+ {
+ "average": [
+ -0.17840398,
+ 0.00960978666666667,
+ -0.14545916333333334,
+ 0.41671050333333337,
+ -0.11476575433333332,
+ -0.005306495043333334,
+ -0.06507552800000001,
+ -0.041544396666666664,
+ 0.085892911,
+ -0.1788737366666667
+ ]
+ },
+ {
+ "EOF": true,
+ "RESPONSE_TIME": 129
+ }
+ ]
+ }
+}
+```
+
+====
+
+
+// Solr URL examples below all have [ and ] characters which, when used with Curl, causes encoding issues so just putting plain http links
+
+Search for the top 10 movies most similar to the target vector that we previously calculated (KNN Query for recommendation):
+
+ 'http://localhost:8983/solr/films/query?q={%21knn%20f=film_vector%20topK=10}[-0.1784,0.0096,-0.1455,0.4167,-0.1148,-0.0053,-0.0651,-0.0415,0.0859,-0.1789]'
+
+* Notice that among the results, there are some animation family movies, such as _Curious George_ and _Bambi_, which makes sense, since the target vector was created with two other animation family movies (_Finding Nemo_ and _Bee Movie_).
+* We also notice that among the results there are two movies that the person already watched. In the next example we will filter them out.
+
+Search for the top 10 movies most similar to the resulting vector, excluding the movies already watched (KNN query with Filter Query):
+
+ http://localhost:8983/solr/films/query?q={!knn%20f=film_vector%20topK=10}[-0.1784,0.0096,-0.1455,0.4167,-0.1148,-0.0053,-0.0651,-0.0415,0.0859,-0.1789]&fq=-id:("%2Fen%2Ffinding_nemo"%20"%2Fen%2Fbee_movie"%20"%2Fen%2Fharry_potter_and_the_chamber_of_secrets_2002")
+
+ - Search for movies with "cinderella" in the name among the top 50 movies most similar to the target vector (KNN as Filter Query):
+
+ http://localhost:8983/solr/films/query?q=name:cinderella&fq={!knn%20f=film_vector%20topK=50}[-0.1784,0.0096,-0.1455,0.4167,-0.1148,-0.0053,-0.0651,-0.0415,0.0859,-0.1789]
+
+ * There are 3 "cinderella" movies in the index, but only 1 is among the top 50 most similar to the target vector (_Cinderella III: A Twist in Time_).
+
+ - Search for movies with "animation" in the genre, and rerank the top 5 documents by combining (sum) the original query score with twice (2x) the similarity to the target vector (KNN with ReRanking):
+
+ http://localhost:8983/solr/films/query?q=genre:animation&rqq={!knn%20f=film_vector%20topK=10000}[-0.1784,0.0096,-0.1455,0.4167,-0.1148,-0.0053,-0.0651,-0.0415,0.0859,-0.1789]&rq={!rerank%20reRankQuery=$rqq%20reRankDocs=5%20reRankWeight=2}
+
+ * To guarantee we calculate the vector similarity score for all the movies, we set `topK=10000`, a number higher than the total number of documents (`1100`).
+
+ * It's possible to combine the vector similarity scores with other scores, by using Sub-query,
+ xref:query-guide:function-queries.adoc[Function Queries] and xref:query-guide:local-params.adoc#parameter-dereferencing[Parameter Dereferencing] Solr features:
+
+ - Search for "harry potter" movies, ranking the results by the similarity to the target vector instead of the lexical query score. Beside the `q` parameter, we define a "sub-query" named `q_vector`, that will calculate the similarity score between all the movies (since we set `topK=10000`). Then we use the sub-query parameter name as input for the `sort`, specifying that we want to rank descending according to the vector similarity score (`sort=$q_vector desc`):
+
+ http://localhost:8983/solr/films/query?q=name:"harry%20potter"&q_vector={!knn%20f=film_vector%20topK=10000}[-0.1784,0.0096,-0.1455,0.4167,-0.1148,-0.0053,-0.0651,-0.0415,0.0859,-0.1789]&sort=$q_vector%20desc
+
+ - Search for movies with "the" in the name, keeping the original lexical query ranking, but returning only movies with similarity to the target vector of 0.8 or higher. Like previously, we define the sub-query `q_vector`, but this time we use it as input for the `frange` filter, specifying that we want documents with at least 0.8 of vector similarity score:
+
+ http://localhost:8983/solr/films/query?q=name:the&q_vector={!knn%20f=film_vector%20topK=10000}[-0.1784,0.0096,-0.1455,0.4167,-0.1148,-0.0053,-0.0651,-0.0415,0.0859,-0.1789]&fq={!frange%20l=0.8}$q_vector
+
+ - Search for "batman" movies, ranking the results by combining 70% of the original lexical query score and 30% of the similarity to the target vector. Besides the `q` main query and the `q_vector` sub-query, we also specify the `q_lexical` query, which will hold the lexical score of the main `q` query. Then we specify a parameter variable called `score_combined`, which scales the lexical and similarity scores, applies the 0.7 and 0.3 weights, then sum the result. We set the `sort` p [...]
+
+ http://localhost:8983/solr/films/query?q=name:batman&q_lexical={!edismax%20v=$q}&q_vector={!knn%20f=film_vector%20topK=10000}[-0.1784,0.0096,-0.1455,0.4167,-0.1148,-0.0053,-0.0651,-0.0415,0.0859,-0.1789]&score_combined=sum(mul(scale($q_lexical,0,1),0.7),mul(scale($q_vector,0,1),0.3))&sort=$score_combined%20desc&fl=name,score,$q_lexical,$q_vector,$score_combined
+
+
+=== Exercise 5 Wrap Up
+
+In this exercise, we used the Schema API to add the vector field, and then learned how to index and query Solr using vector data structure.