You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "shubhamvishu (via GitHub)" <gi...@apache.org> on 2024/03/18 06:22:15 UTC
[PR] Pass custom similarity function to similarityToQueryVector API [lucene]
shubhamvishu opened a new pull request, #13187:
URL: https://github.com/apache/lucene/pull/13187
### Description
This PR allows passing a custom vector similarity function to DVS implementations of `VectorSimilarityValuesSource` as opposed to current behaviour which by default only allows to use the configured `VectorSimilarityFunction` associated with the KNN/vector field.
**Tests :** Added some unit tests
<!--
If this is your first contribution to Lucene, please make sure you have reviewed the contribution guide.
https://github.com/apache/lucene/blob/main/CONTRIBUTING.md
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Pass custom similarity function to similarityToQueryVector API [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #13187:
URL: https://github.com/apache/lucene/pull/13187#issuecomment-2004052748
> Though I'm not sure if this change conflicts with or makes things difficult for the ongoing efforts to have pluggability (maybe @benwtrent would be interested in sharing his thoughts?).
I would assume any change here would want to take advantage of any pluggable similarity. If so, this change would 100% be effected.
What makes this PR doubly worrying is that none of the interfaces are marked experimental. Meaning, if its released and we want to change it, we gotta be compatible, that doesn't seem wise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Pass custom similarity function to similarityToQueryVector API [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on PR #13187:
URL: https://github.com/apache/lucene/pull/13187#issuecomment-2003799323
There is some discussion how to make similarities more pluggable https://github.com/apache/lucene/issues/13182 that seems relevant. Part of the idea there is to accept ordinal values rather than arrays so that internally we can make use of directory memory access rather than having to materialize values on the heap. This issue could still fit into that framework I think, just providing a query vector but not a document vector? I guess we should think about whether this change would lock us in further to the current situation or not. Maybe we could go ahead with this and then change it later when that other issue lands? But it could make it more difficult to "plug in" implementations if they have to be reimplemented
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Pass custom similarity function to similarityToQueryVector API [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13187:
URL: https://github.com/apache/lucene/pull/13187#issuecomment-2060105283
This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Pass custom similarity function to similarityToQueryVector API [lucene]
Posted by "shubhamvishu (via GitHub)" <gi...@apache.org>.
shubhamvishu commented on PR #13187:
URL: https://github.com/apache/lucene/pull/13187#issuecomment-2004568222
@benwtrent So should we instead wait for the pluggability support and discard this for now? or Is it possible to go forward with this?
> What makes this PR doubly worrying is that none of the interfaces are marked experimental. Meaning, if its released and we want to change it, we gotta be compatible, that doesn't seem wise.
I'm not sure if we can mark them experimental at this point but if yes, maybe we could first mark these classes as experimental so that we don't have to worry about backward compatibility later?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Pass custom similarity function to similarityToQueryVector API [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #13187:
URL: https://github.com/apache/lucene/pull/13187#issuecomment-2031759399
I am sorry this work has stalled. But I have been iterating on https://github.com/apache/lucene/pull/13200 for a week now. Its getting to a palatable place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Pass custom similarity function to similarityToQueryVector API [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13187:
URL: https://github.com/apache/lucene/pull/13187#issuecomment-2030835653
This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Pass custom similarity function to similarityToQueryVector API [lucene]
Posted by "shubhamvishu (via GitHub)" <gi...@apache.org>.
shubhamvishu commented on PR #13187:
URL: https://github.com/apache/lucene/pull/13187#issuecomment-2003991803
Thanks for the review @msokolov! The idea to make it pluggable seems relevant and interesting. Currently it was not possible to use any custom vector similarity function other than the configured one which for instance in case of `DOT_PRODUCT` similarity gives the scaled score and not the actual raw dot product. So by that logic I think this change would be good.
> I guess we should think about whether this change would lock us in further to the current situation or not. Maybe we could go ahead with this and then change it later when that other issue lands? But it could make it more difficult to "plug in" implementations if they have to be reimplemented
Yes, we should definitely take that into account. Though I'm not sure if this change conflicts with or makes things difficult for the ongoing efforts to have pluggability (maybe @benwtrent would be interested in sharing his thoughts?).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Pass custom similarity function to similarityToQueryVector API [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13187:
URL: https://github.com/apache/lucene/pull/13187#issuecomment-3408725365
This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org