You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "benwtrent (via GitHub)" <gi...@apache.org> on 2025/01/29 14:03:34 UTC
[I] Multi-threaded vector search over multiple segments can lead to inconsistent results [lucene]
benwtrent opened a new issue, #14180:
URL: https://github.com/apache/lucene/issues/14180
### Description
Related to: https://github.com/apache/lucene/pull/14167
But multi-threaded search over multiple segments in addition to multi-leaf collection (e.g. information sharing) can lead to consistent results at low `k` values.
It is possible to get more consistent results, and likely remove most inconsistencies by simply gathering more neighbors (`k` in the query, `fanout` in lucene util, or `efSearch` if you will...). However, while HNSW search is approximate, we should strive for consistency.
### Version and environment details
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Multi-threaded vector search over multiple segments can lead to inconsistent results [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on issue #14180:
URL: https://github.com/apache/lucene/issues/14180#issuecomment-2863036189
> Has the problem been resolved? Will the next version of Elasticsearch fix this issue?
It has been resolved in Lucene. There is no set fix version for anything that depends on Lucene, like Elasticsearch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Multi-threaded vector search over multiple segments can lead to inconsistent results [lucene]
Posted by "tteofili (via GitHub)" <gi...@apache.org>.
tteofili commented on issue #14180:
URL: https://github.com/apache/lucene/issues/14180#issuecomment-2730194375
> Could you please let me know which future version of Elasticsearch will resolve the vector search consistency problem?
we are investigating on a proper solution to this issue that addresses the determinism while retaining good efficiency (e.g., see some early experiments in this [draft PR](https://github.com/apache/lucene/pull/14191))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Multi-threaded vector search over multiple segments can lead to inconsistent results [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #14180:
URL: https://github.com/apache/lucene/issues/14180#issuecomment-2894399826
It sounds like this is fixed, I will close this now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Multi-threaded vector search over multiple segments can lead to inconsistent results [lucene]
Posted by "Zona-hu (via GitHub)" <gi...@apache.org>.
Zona-hu commented on issue #14180:
URL: https://github.com/apache/lucene/issues/14180#issuecomment-2728315199
> ### 描述
> 相关：[#14167](https://github.com/apache/lucene/pull/14167)
>
> 但是，除了多叶收集（例如信息共享）之外，对多个段进行多线程搜索也可以在低值下获得一致的结果`k`。
>
> 有可能获得更一致的结果，并且可能通过简单地收集更多邻居（`k`在查询中、`fanout`在 lucene 工具中，或者`efSearch`如果你愿意的话......）来消除大多数不一致性。然而，虽然 HNSW 搜索是近似的，但我们应该努力保持一致性。
>
> ### 版本和环境详细信息
> _没有回应_
Hello, I filed this issue:
https://github.com/apache/lucene/issues/14180
Could you please let me know which future version of Elasticsearch will resolve the vector search consistency problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Multi-threaded vector search over multiple segments can lead to inconsistent results [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand closed issue #14180: Multi-threaded vector search over multiple segments can lead to inconsistent results
URL: https://github.com/apache/lucene/issues/14180
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Multi-threaded vector search over multiple segments can lead to inconsistent results [lucene]
Posted by "tteofili (via GitHub)" <gi...@apache.org>.
tteofili commented on issue #14180:
URL: https://github.com/apache/lucene/issues/14180#issuecomment-2854021698
this should be now fixed via https://github.com/apache/lucene/pull/14226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Multi-threaded vector search over multiple segments can lead to inconsistent results [lucene]
Posted by "Zona-hu (via GitHub)" <gi...@apache.org>.
Zona-hu commented on issue #14180:
URL: https://github.com/apache/lucene/issues/14180#issuecomment-2861492083
> this should be now fixed via [#14226](https://github.com/apache/lucene/pull/14226)
Has the problem been resolved? Will the next version of Elasticsearch fix this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org