You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2021/04/23 22:25:14 UTC
[GitHub] [lucene] jtibshirani opened a new pull request #104: LUCENE-9908: Move VectorValues#search to LeafReader
jtibshirani opened a new pull request #104:
URL: https://github.com/apache/lucene/pull/104
This PR removes `VectorValues#search` in favor of exposing NN search through
`VectorReader#search` and `LeafReader#searchNearestVectors`. It also marks the
vector methods on `LeafReader` as experimental.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jtibshirani merged pull request #104: LUCENE-9908: Move VectorValues#search to LeafReader
Posted by GitBox <gi...@apache.org>.
jtibshirani merged pull request #104:
URL: https://github.com/apache/lucene/pull/104
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jtibshirani commented on a change in pull request #104: LUCENE-9908: Move VectorValues#search to LeafReader
Posted by GitBox <gi...@apache.org>.
jtibshirani commented on a change in pull request #104:
URL: https://github.com/apache/lucene/pull/104#discussion_r619525800
##########
File path: lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90VectorReader.java
##########
@@ -199,40 +229,53 @@ public void checkIntegrity() throws IOException {
@Override
public VectorValues getVectorValues(String field) throws IOException {
- FieldInfo info = fieldInfos.fieldInfo(field);
- if (info == null) {
+ FieldEntry fieldEntry = fields.get(field);
Review comment:
Across the different reader implementations, we aren't that consistent when we return `null` vs. empty. For now I just matched the current pattern in each reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jtibshirani commented on a change in pull request #104: LUCENE-9908: Move VectorValues#search to LeafReader
Posted by GitBox <gi...@apache.org>.
jtibshirani commented on a change in pull request #104:
URL: https://github.com/apache/lucene/pull/104#discussion_r620479333
##########
File path: lucene/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java
##########
@@ -1357,6 +1359,11 @@ public VectorValues getVectorValues(String fieldName) {
return VectorValues.EMPTY;
}
+ @Override
+ public TopDocs searchNearestVectors(String field, float[] target, int k, int fanout) {
+ return TopDocsCollector.EMPTY_TOPDOCS;
Review comment:
This makes sense to me. I'll update this and `getVectorValues` above to return `null`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jpountz commented on a change in pull request #104: LUCENE-9908: Move VectorValues#search to LeafReader
Posted by GitBox <gi...@apache.org>.
jpountz commented on a change in pull request #104:
URL: https://github.com/apache/lucene/pull/104#discussion_r620057599
##########
File path: lucene/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java
##########
@@ -1357,6 +1359,11 @@ public VectorValues getVectorValues(String fieldName) {
return VectorValues.EMPTY;
}
+ @Override
+ public TopDocs searchNearestVectors(String field, float[] target, int k, int fanout) {
+ return TopDocsCollector.EMPTY_TOPDOCS;
Review comment:
Should it be `null`? My understanding based on the rest of the PR is that we use `null` when a field doesn't have vectors and `EMPTY_TOP_DOCS` when a field has vectors but not on the current segment. Since MemoryIndex doesn't support indexing vectors should it return null?
##########
File path: lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90VectorReader.java
##########
@@ -199,40 +229,53 @@ public void checkIntegrity() throws IOException {
@Override
public VectorValues getVectorValues(String field) throws IOException {
- FieldInfo info = fieldInfos.fieldInfo(field);
- if (info == null) {
+ FieldEntry fieldEntry = fields.get(field);
Review comment:
Ideally we'd be consistent with what we do with doc values:
- The low-level API on `VectorReader` only supports getting data for fields that exist and have vectors enabled on their FieldInfo, the behavior is undefined otherwise.
- The high-level API on `LeafReader` checks the FieldInfo and returns `null` for fields that do not exist or that don't have vectors enabled, and otherwise delegates to the `VectorReader`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] jtibshirani commented on a change in pull request #104: LUCENE-9908: Move VectorValues#search to LeafReader
Posted by GitBox <gi...@apache.org>.
jtibshirani commented on a change in pull request #104:
URL: https://github.com/apache/lucene/pull/104#discussion_r620480117
##########
File path: lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90VectorReader.java
##########
@@ -199,40 +229,53 @@ public void checkIntegrity() throws IOException {
@Override
public VectorValues getVectorValues(String field) throws IOException {
- FieldInfo info = fieldInfos.fieldInfo(field);
- if (info == null) {
+ FieldEntry fieldEntry = fields.get(field);
Review comment:
I'll try to fix this before merging (for both vector-related methods).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org