You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Joel Bernstein (Jira)" <ji...@apache.org> on 2019/11/19 18:49:00 UTC
[jira] [Comment Edited] (SOLR-12890) Vector Search in Solr
(Umbrella Issue)
[ https://issues.apache.org/jira/browse/SOLR-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948916#comment-16948916 ]
Joel Bernstein edited comment on SOLR-12890 at 11/19/19 6:48 PM:
-----------------------------------------------------------------
h1. Rough survey of some available approaches:
h2. 1) Vector Scoring using Streaming Expressions (works now):
*Docs:* (paste each into "Documents" pane in Solr Admin UI as type:"json")
{code:java}
curl -X POST -H "Content-Type: application/json" http://localhost:8983/solr/food_collection/update?commit=true --data-binary '
[
{"id": "1", "name_s":"donut","vector_fs":[5.0,0.0,1.0,5.0,0.0,4.0,5.0,1.0]},
{"id": "2", "name_s":"apple juice","vector_fs":[1.0,5.0,0.0,0.0,0.0,4.0,4.0,3.0]},
{"id": "3", "name_s":"cappuccino","vector_fs":[0.0,5.0,3.0,0.0,4.0,1.0,2.0,3.0]},
{"id": "4", "name_s":"cheese pizza","vector_fs":[5.0,0.0,4.0,4.0,0.0,1.0,5.0,2.0]},
{"id": "5", "name_s":"green tea","vector_fs":[0.0,5.0,0.0,0.0,2.0,1.0,1.0,5.0]},
{"id": "6", "name_s":"latte","vector_fs":[0.0,5.0,4.0,0.0,4.0,1.0,3.0,3.0]},
{"id": "7", "name_s":"soda","vector_fs":[0.0,5.0,0.0,0.0,3.0,5.0,5.0,0.0]},
{"id": "8", "name_s":"cheese bread sticks","vector_fs":[5.0,0.0,4.0,5.0,0.0,1.0,4.0,2.0]},
{"id": "9", "name_s":"water","vector_fs":[0.0,5.0,0.0,0.0,0.0,0.0,0.0,5.0]},
{"id": "10", "name_s":"cinnamon bread sticks","vector_fs":[5.0,0.0,1.0,5.0,0.0,3.0,4.0,2.0]}
]{code}
*Streaming Expression:*
{code:java}
sort(
select(
search(food_collection,
q="*:*",
fl="id,vector_fs",
sort="id asc", rows=3),
cosineSimilarity(vector_fs, array(5.1,0.0,1.0,5.0,0.0,4.0,5.0,1.0)) as sim, id),
by="sim desc")
*Response:*
{
"result-set": {
"docs": [
{ "sim": 0.99996111, "id": "1" },
{ "sim": 0.98590279, "id": "10" },
{ "sim": 0.55566643, "id": "2" },
{ "EOF": true, "RESPONSE_TIME": 10 }
]
}
}{code}
*Benefits*:
1) Works Now out of the box
*Drawbacks*:
1) Have to switch searches to using Streaming Expressions, which may not be practical in some use cases.
2) Solr doesn't have multi-dimensional point field support yet (SOLR-11077), so you can only store one vector per field per document.
3) Requires traversing all vectors and scoring them. Needs some sore of KNN option (possibly this could be done with another inner-streaming expression using a hash function?)
h2. 2) Available Solr Vector Search Plugin (works now):
[https://github.com/saaay71/solr-vector-scoring]
Note: I recently reached out to Ali (the author of this plugin) and asked him to add an ASL 2.0 license, which he has now done, so we can pull in this code as needed.
*Docs*
{code:java}
curl -X POST -H "Content-Type: application/json" http://localhost:8983/solr/{your-collection-name}/update?commit=true --data-binary '
[
{"name":"example 0", "vector":"0|1.55 1|3.53 2|2.3 3|0.7 4|3.44 5|2.33 "},
{"name":"example 1", "vector":"0|3.54 1|0.4 2|4.16 3|4.88 4|4.28 5|4.25 "},
{"name":"example 2", "vector":"0|1.11 1|0.6 2|1.47 3|1.99 4|2.91 5|1.01 "},
{"name":"example 3", "vector":"0|0.06 1|4.73 2|0.29 3|1.27 4|0.69 5|3.9 "},
{"name":"example 4", "vector":"0|4.01 1|3.69 2|2 3|4.36 4|1.09 5|0.1 "},
{"name":"example 5", "vector":"0|0.64 1|3.95 2|1.03 3|1.65 4|0.99 5|0.09 "}
]'
{code}
*Request:*
{code:java}
http://localhost:8983/solr/{your-collection-name}/query?fl=name,score,vector&q={!vp f=vector vector="0.1,4.75,0.3,1.2,0.7,4.0"}
{code}
*Response:*
{code:java}
{
"responseHeader":{
"status":0,
"QTime":1,
"params":{
"q":"{!myqp f=vector vector=\"0.1,4.75,0.3,1.2,0.7,4.0\"}",
"fl":"name,score,vector"}},
"response":{"numFound":6,"start":0,"maxScore":0.99984086,"docs":[
{
"name":["example 3"],
"vector":["0|0.06 1|4.73 2|0.29 3|1.27 4|0.69 5|3.9 "],
"score":0.99984086},
{
"name":["example 0"],
"vector":["0|1.55 1|3.53 2|2.3 3|0.7 4|3.44 5|2.33 "],
"score":0.7693964},
{
"name":["example 5"],
"vector":["0|0.64 1|3.95 2|1.03 3|1.65 4|0.99 5|0.09 "],
"score":0.76322395},
{
"name":["example 4"],
"vector":["0|4.01 1|3.69 2|2 3|4.36 4|1.09 5|0.1 "],
"score":0.5328145},
{
"name":["example 1"],
"vector":["0|3.54 1|0.4 2|4.16 3|4.88 4|4.28 5|4.25 "],
"score":0.48513117},
{
"name":["example 2"],
"vector":["0|1.11 1|0.6 2|1.47 3|1.99 4|2.91 5|1.01 "],
"score":0.44909418}]
}}
{code}
*Benefits:*
1) Works now (when you install the plugin)
2) Can use in regular search (Search Handler) and don't have to switch to streaming expressions
*Drawbacks*:
1) Slow implementation. It uses payloads to store the values within the vector, and traversing those is very expensive. If we were going to follow this approach, at a minimum we should switch from using payloads to overriding term frequencies for a speedup.
2) Only supports one vector per field per document (I think... haven't tried with a multi-valued text field, but it looks like the payload scoring logic expects only a single value per dimension. Might be possible to modify this...)
3) Requires traversing all vectors and scoring them. Needs some sort of KNN option to not be slow at scale.
h2. 3) Available Solr Vector Search Plugin with LSH Hashing (works now)
[https://github.com/moshebla/solr-vector-scoring]
[~moshebla], who created this JIRA issue, forked option #2 above and added and LSH implementation so that a KNN filter can be applied before the vector scoring, greatly improving the speed of the vector search / scoring.
*Docs:*
{code:java}
curl -X POST -H "Content-Type: application/json" http://localhost:8983/solr/{your-collection-name}/update?update.chain=LSH&commit=true --data-binary '
[
{"id":"1", "vector":"1.55,3.53,2.3,0.7,3.44,2.33"},
{"id":"2", "vector":"3.54,0.4,4.16,4.88,4.28,4.25"}
]'
{code}
*Request:*
{code:java}
http://localhost:8983/solr/{your-collection-name}/query?fl=name,score,vector&q={!vp f=vector vector=\"1.55,3.53,2.3,0.7,3.44,2.33\" lsh=\"true\" reRankDocs=\"5\"}&fl=name,score,vector,_vector_,_lsh_hash_
{code}
*Response:*
{code:java}
{
"responseHeader":{
"status":0,
"QTime":8,
"params":{
"q":"{!vp f=vector vector=\"1.55,3.53,2.3,0.7,3.44,2.33\" lsh=\"true\" reRankDocs=\"5\"}",
"fl":"id, score, vector, _vector_, _lsh_hash_",
"wt":"xml"}},
"response":{"numFound":1,"start":0,"maxScore":36.65736,"docs":[
{
"id": "1",
"vector":"1.55,3.53,2.3,0.7,3.44,2.33",
"_vector_":"/z/GZmZAYeuFQBMzMz8zMzNAXCj2QBUeuA==",
"_lsh_hash_":["0_8",
"1_35",
"2_7",
"3_10",
"4_2",
"5_35",
"6_16",
"7_30",
"8_27",
"9_12",
"10_7",
"11_32",
"12_48",
"13_36",
"14_10",
"15_7",
"16_42",
"17_5",
"18_3",
"19_2",
"20_1",
"21_0",
"22_24",
"23_18",
"24_42",
"25_31",
"26_35",
"27_8",
"28_1",
"29_24",
"30_47",
"31_14",
"32_22",
"33_39",
"34_0",
"35_34",
"36_34",
"37_39",
"38_27",
"39_27",
"40_45",
"41_10",
"42_21",
"43_34",
"44_41",
"45_9",
"46_31",
"47_0",
"48_4",
"49_43"],
"score":36.65736}
]
}
}
{code}
*Benefits:*
1) Works now (when you install the plugin)
2) Can use in regular search (Search Handler) and don't have to switch to streaming expressions
3) Provides a KNN implementation that prevents every document's vector from having to be scored
*Drawbacks:*
1) Slow. Even though the KNN reduces unnecessarily scoring all vectors, using payloads in still inefficient here. If we were going to follow this approach, at a minimum we should switch from using payloads to overriding term frequencies for a speedup.
2) Only supports one vector per field per document (same as original it was forked from)
3) Doesn't currently have an ASL 2.0 license on it for reuse. [~moshebla] - since Ali has added a license to the original repo now, can you please pull that into your repo so that your changes are also covered under ASL2.0? Thanks!
h2. 4) Port over the Elasticsearch implementation
Elasticsearch recently implemented sparse and dense vector fields, though they chose to release the feature under their proprietary Elastic license instead of an open source license. HOWEVER, when they were originally implementing this feature they were intending for it to be open source, and only later decided to change the license to be proprietary, so most of the feature was built under and ASL 2.0 license before they restricted it. This means that we can port over any part of their implementation that existed prior to this commit under the ASL 2.0 license: [https://github.com/elastic/elasticsearch/commit/952ddf247a2df8ade64ae067c1904436fd7a2ba8]
*Benefits*:
1) Encoded vectors into BinaryDocValues, so certainly more efficient that the approaches with the Solr Plugins above (using payloads)
*Drawbacks:*
1) Doesn't work with Solr yet, so would have to do more work to port it over.
2) Can't copy over future improvements since Elasticsearch isn't open source (only early versions of this feature were)
3) Only supports one vector per field per document currently (though this shouldn't be too hard to change)
4) Doesn't appear to provide a quantized representation for KNN filtering prior to scoring, so likely slow at scale.
h2. 5) Port over Open Distro for Stretchysearch implementation
[https://github.com/opendistro-for-elasticsearch/k-NN]
Open Distro for "Stretchysearch" (not using the original project name because Elastic is suing them for Trademark infringement for doing so) created their own ASL2.0 plugin for their Open Distro version which implements vector search with KNN support. On the surface, this looks like it may be an enhanced version over the Elastic version in #4 above, but it appears to still be a work in progress, so not sure if it is production ready yet (haven't tried it).
*Benefits:*
1) Provides a quantized representation of the vectors for KNN, so should still be fast at scale with lots of docs
2) They appear to making codec-level changes, which implies this approach may end up being way more efficient than any of the others mentioned above over time.
*Drawbacks:*
1) Developed against "Stretchysearch" code base, so will take extra effort to port to Solr.
2) Looks like a work in progress, so likely not ready for production use and porting
3) Only supports one vector per field per document currently
h2. 6) Others please contribute ideas!
There's lots of ways to approach this problem, and we have some really smart people in this community. All ideas welcome - possibly we should even implement a few different approaches.
was (Author: solrtrey):
h1. Rough survey of some available approaches:
h2. 1) Vector Scoring using Streaming Expressions (works now):
*Docs:* (paste each into "Documents" pane in Solr Admin UI as type:"json")
{code:java}
curl -X POST -H "Content-Type: application/json" http://localhost:8983/solr/food_collection/update?commit=true --data-binary '
[
{"id": "1", "name_s":"donut","vector_fs":[5.0,0.0,1.0,5.0,0.0,4.0,5.0,1.0]},
{"id": "2", "name_s":"apple juice","vector_fs":[1.0,5.0,0.0,0.0,0.0,4.0,4.0,3.0]},
{"id": "3", "name_s":"cappuccino","vector_fs":[0.0,5.0,3.0,0.0,4.0,1.0,2.0,3.0]},
{"id": "4", "name_s":"cheese pizza","vector_fs":[5.0,0.0,4.0,4.0,0.0,1.0,5.0,2.0]},
{"id": "5", "name_s":"green tea","vector_fs":[0.0,5.0,0.0,0.0,2.0,1.0,1.0,5.0]},
{"id": "6", "name_s":"latte","vector_fs":[0.0,5.0,4.0,0.0,4.0,1.0,3.0,3.0]},
{"id": "7", "name_s":"soda","vector_fs":[0.0,5.0,0.0,0.0,3.0,5.0,5.0,0.0]},
{"id": "8", "name_s":"cheese bread sticks","vector_fs":[5.0,0.0,4.0,5.0,0.0,1.0,4.0,2.0]},
{"id": "9", "name_s":"water","vector_fs":[0.0,5.0,0.0,0.0,0.0,0.0,0.0,5.0]},
{"id": "10", "name_s":"cinnamon bread sticks","vector_fs":[5.0,0.0,1.0,5.0,0.0,3.0,4.0,2.0]}
]{code}
*Streaming Expression:*
{code:java}
sort(select(search(food_collection, q="*:*", fl="id,vector_fs", sort="id asc", rows=3), cosineSimilarity(vector_fs, array(5.1,0.0,1.0,5.0,0.0,4.0,5.0,1.0)) as sim, id), by="sim desc")\{code}
*Response:*
{
"result-set": {
"docs": [
{ "sim": 0.99996111, "id": "1" },
{ "sim": 0.98590279, "id": "10" },
{ "sim": 0.55566643, "id": "2" },
{ "EOF": true, "RESPONSE_TIME": 10 }
]
}
}{code}
*Benefits*:
1) Works Now out of the box
*Drawbacks*:
1) Have to switch searches to using Streaming Expressions, which may not be practical in some use cases.
2) Solr doesn't have multi-dimensional point field support yet (SOLR-11077), so you can only store one vector per field per document.
3) Requires traversing all vectors and scoring them. Needs some sore of KNN option (possibly this could be done with another inner-streaming expression using a hash function?)
h2. 2) Available Solr Vector Search Plugin (works now):
[https://github.com/saaay71/solr-vector-scoring]
Note: I recently reached out to Ali (the author of this plugin) and asked him to add an ASL 2.0 license, which he has now done, so we can pull in this code as needed.
*Docs*
{code:java}
curl -X POST -H "Content-Type: application/json" http://localhost:8983/solr/{your-collection-name}/update?commit=true --data-binary '
[
{"name":"example 0", "vector":"0|1.55 1|3.53 2|2.3 3|0.7 4|3.44 5|2.33 "},
{"name":"example 1", "vector":"0|3.54 1|0.4 2|4.16 3|4.88 4|4.28 5|4.25 "},
{"name":"example 2", "vector":"0|1.11 1|0.6 2|1.47 3|1.99 4|2.91 5|1.01 "},
{"name":"example 3", "vector":"0|0.06 1|4.73 2|0.29 3|1.27 4|0.69 5|3.9 "},
{"name":"example 4", "vector":"0|4.01 1|3.69 2|2 3|4.36 4|1.09 5|0.1 "},
{"name":"example 5", "vector":"0|0.64 1|3.95 2|1.03 3|1.65 4|0.99 5|0.09 "}
]'
{code}
*Request:*
{code:java}
http://localhost:8983/solr/{your-collection-name}/query?fl=name,score,vector&q={!vp f=vector vector="0.1,4.75,0.3,1.2,0.7,4.0"}
{code}
*Response:*
{code:java}
{
"responseHeader":{
"status":0,
"QTime":1,
"params":{
"q":"{!myqp f=vector vector=\"0.1,4.75,0.3,1.2,0.7,4.0\"}",
"fl":"name,score,vector"}},
"response":{"numFound":6,"start":0,"maxScore":0.99984086,"docs":[
{
"name":["example 3"],
"vector":["0|0.06 1|4.73 2|0.29 3|1.27 4|0.69 5|3.9 "],
"score":0.99984086},
{
"name":["example 0"],
"vector":["0|1.55 1|3.53 2|2.3 3|0.7 4|3.44 5|2.33 "],
"score":0.7693964},
{
"name":["example 5"],
"vector":["0|0.64 1|3.95 2|1.03 3|1.65 4|0.99 5|0.09 "],
"score":0.76322395},
{
"name":["example 4"],
"vector":["0|4.01 1|3.69 2|2 3|4.36 4|1.09 5|0.1 "],
"score":0.5328145},
{
"name":["example 1"],
"vector":["0|3.54 1|0.4 2|4.16 3|4.88 4|4.28 5|4.25 "],
"score":0.48513117},
{
"name":["example 2"],
"vector":["0|1.11 1|0.6 2|1.47 3|1.99 4|2.91 5|1.01 "],
"score":0.44909418}]
}}
{code}
*Benefits:*
1) Works now (when you install the plugin)
2) Can use in regular search (Search Handler) and don't have to switch to streaming expressions
*Drawbacks*:
1) Slow implementation. It uses payloads to store the values within the vector, and traversing those is very expensive. If we were going to follow this approach, at a minimum we should switch from using payloads to overriding term frequencies for a speedup.
2) Only supports one vector per field per document (I think... haven't tried with a multi-valued text field, but it looks like the payload scoring logic expects only a single value per dimension. Might be possible to modify this...)
3) Requires traversing all vectors and scoring them. Needs some sort of KNN option to not be slow at scale.
h2. 3) Available Solr Vector Search Plugin with LSH Hashing (works now)
[https://github.com/moshebla/solr-vector-scoring]
[~moshebla], who created this JIRA issue, forked option #2 above and added and LSH implementation so that a KNN filter can be applied before the vector scoring, greatly improving the speed of the vector search / scoring.
*Docs:*
{code:java}
curl -X POST -H "Content-Type: application/json" http://localhost:8983/solr/{your-collection-name}/update?update.chain=LSH&commit=true --data-binary '
[
{"id":"1", "vector":"1.55,3.53,2.3,0.7,3.44,2.33"},
{"id":"2", "vector":"3.54,0.4,4.16,4.88,4.28,4.25"}
]'
{code}
*Request:*
{code:java}
http://localhost:8983/solr/{your-collection-name}/query?fl=name,score,vector&q={!vp f=vector vector=\"1.55,3.53,2.3,0.7,3.44,2.33\" lsh=\"true\" reRankDocs=\"5\"}&fl=name,score,vector,_vector_,_lsh_hash_
{code}
*Response:*
{code:java}
{
"responseHeader":{
"status":0,
"QTime":8,
"params":{
"q":"{!vp f=vector vector=\"1.55,3.53,2.3,0.7,3.44,2.33\" lsh=\"true\" reRankDocs=\"5\"}",
"fl":"id, score, vector, _vector_, _lsh_hash_",
"wt":"xml"}},
"response":{"numFound":1,"start":0,"maxScore":36.65736,"docs":[
{
"id": "1",
"vector":"1.55,3.53,2.3,0.7,3.44,2.33",
"_vector_":"/z/GZmZAYeuFQBMzMz8zMzNAXCj2QBUeuA==",
"_lsh_hash_":["0_8",
"1_35",
"2_7",
"3_10",
"4_2",
"5_35",
"6_16",
"7_30",
"8_27",
"9_12",
"10_7",
"11_32",
"12_48",
"13_36",
"14_10",
"15_7",
"16_42",
"17_5",
"18_3",
"19_2",
"20_1",
"21_0",
"22_24",
"23_18",
"24_42",
"25_31",
"26_35",
"27_8",
"28_1",
"29_24",
"30_47",
"31_14",
"32_22",
"33_39",
"34_0",
"35_34",
"36_34",
"37_39",
"38_27",
"39_27",
"40_45",
"41_10",
"42_21",
"43_34",
"44_41",
"45_9",
"46_31",
"47_0",
"48_4",
"49_43"],
"score":36.65736}
]
}
}
{code}
*Benefits:*
1) Works now (when you install the plugin)
2) Can use in regular search (Search Handler) and don't have to switch to streaming expressions
3) Provides a KNN implementation that prevents every document's vector from having to be scored
*Drawbacks:*
1) Slow. Even though the KNN reduces unnecessarily scoring all vectors, using payloads in still inefficient here. If we were going to follow this approach, at a minimum we should switch from using payloads to overriding term frequencies for a speedup.
2) Only supports one vector per field per document (same as original it was forked from)
3) Doesn't currently have an ASL 2.0 license on it for reuse. [~moshebla] - since Ali has added a license to the original repo now, can you please pull that into your repo so that your changes are also covered under ASL2.0? Thanks!
h2. 4) Port over the Elasticsearch implementation
Elasticsearch recently implemented sparse and dense vector fields, though they chose to release the feature under their proprietary Elastic license instead of an open source license. HOWEVER, when they were originally implementing this feature they were intending for it to be open source, and only later decided to change the license to be proprietary, so most of the feature was built under and ASL 2.0 license before they restricted it. This means that we can port over any part of their implementation that existed prior to this commit under the ASL 2.0 license: [https://github.com/elastic/elasticsearch/commit/952ddf247a2df8ade64ae067c1904436fd7a2ba8]
*Benefits*:
1) Encoded vectors into BinaryDocValues, so certainly more efficient that the approaches with the Solr Plugins above (using payloads)
*Drawbacks:*
1) Doesn't work with Solr yet, so would have to do more work to port it over.
2) Can't copy over future improvements since Elasticsearch isn't open source (only early versions of this feature were)
3) Only supports one vector per field per document currently (though this shouldn't be too hard to change)
4) Doesn't appear to provide a quantized representation for KNN filtering prior to scoring, so likely slow at scale.
h2. 5) Port over Open Distro for Stretchysearch implementation
[https://github.com/opendistro-for-elasticsearch/k-NN]
Open Distro for "Stretchysearch" (not using the original project name because Elastic is suing them for Trademark infringement for doing so) created their own ASL2.0 plugin for their Open Distro version which implements vector search with KNN support. On the surface, this looks like it may be an enhanced version over the Elastic version in #4 above, but it appears to still be a work in progress, so not sure if it is production ready yet (haven't tried it).
*Benefits:*
1) Provides a quantized representation of the vectors for KNN, so should still be fast at scale with lots of docs
2) They appear to making codec-level changes, which implies this approach may end up being way more efficient than any of the others mentioned above over time.
*Drawbacks:*
1) Developed against "Stretchysearch" code base, so will take extra effort to port to Solr.
2) Looks like a work in progress, so likely not ready for production use and porting
3) Only supports one vector per field per document currently
h2. 6) Others please contribute ideas!
There's lots of ways to approach this problem, and we have some really smart people in this community. All ideas welcome - possibly we should even implement a few different approaches.
> Vector Search in Solr (Umbrella Issue)
> --------------------------------------
>
> Key: SOLR-12890
> URL: https://issues.apache.org/jira/browse/SOLR-12890
> Project: Solr
> Issue Type: New Feature
> Reporter: mosh
> Priority: Major
>
> We have recently come across a need to index documents containing vectors using solr, and have even worked on a small POC. We used an URP to calculate the LSH(we chose to use the superbit algorithm, but the code is designed in a way the algorithm picked can be easily chagned), and stored the vector in either sparse or dense forms, in a binary field.
> Perhaps an addition of an LSH URP in conjunction with a query parser that uses the same properties to calculate LSH(or maybe ktree, or some other algorithm all together) should be considered as a Solr feature?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org