You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by jp...@apache.org on 2022/11/23 10:06:44 UTC
[lucene] branch branch_9x updated (0192d415570 -> 3352bf0a647)
This is an automated email from the ASF dual-hosted git repository.
jpountz pushed a change to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/lucene.git
from 0192d415570 Add back-compat indices for 9.4.2
new 604d320326e Remove VectorValues#EMPTY. (#11961)
new f17df395ab3 No longer announce releases on general@l.a.o. (#11967)
new 3352bf0a647 Enforce VectorValues.cost() is equal to size(). (#11962)
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
dev-tools/scripts/releaseWizard.yaml | 2 +-
lucene/CHANGES.txt | 7 +++
.../lucene90/Lucene90HnswVectorsReader.java | 5 --
.../lucene91/Lucene91HnswVectorsReader.java | 5 --
.../lucene92/OffHeapVectorValues.java | 10 ----
.../simpletext/SimpleTextKnnVectorsReader.java | 8 +--
.../org/apache/lucene/codecs/KnnVectorsFormat.java | 5 +-
.../org/apache/lucene/codecs/KnnVectorsWriter.java | 5 --
.../codecs/lucene94/OffHeapVectorValues.java | 10 ----
.../lucene/index/BufferingKnnVectorsWriter.java | 5 --
.../apache/lucene/index/FilterVectorValues.java | 5 --
.../java/org/apache/lucene/index/VectorValues.java | 58 +++-------------------
.../apache/lucene/util/hnsw/MockVectorValues.java | 5 --
.../org/apache/lucene/util/hnsw/TestHnswGraph.java | 5 --
14 files changed, 18 insertions(+), 117 deletions(-)
[lucene] 03/03: Enforce VectorValues.cost() is equal to size(). (#11962)
Posted by jp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jpountz pushed a commit to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 3352bf0a647390bcc61319dc4c6a04d9bcfd0fc1
Author: Adrien Grand <jp...@gmail.com>
AuthorDate: Wed Nov 23 11:05:00 2022 +0100
Enforce VectorValues.cost() is equal to size(). (#11962)
`VectorValues` have a `cost()` method that reports an approximate number of
documents that have a vector, but also a `size()` method that reports the
accurate number of vectors in the field. Since KNN vectors only support
single-valued fields we should enforce that `cost()` returns the `size()`.
---
lucene/CHANGES.txt | 3 +++
.../lucene90/Lucene90HnswVectorsReader.java | 5 -----
.../lucene91/Lucene91HnswVectorsReader.java | 5 -----
.../backward_codecs/lucene92/OffHeapVectorValues.java | 10 ----------
.../codecs/simpletext/SimpleTextKnnVectorsReader.java | 5 -----
.../java/org/apache/lucene/codecs/KnnVectorsWriter.java | 5 -----
.../apache/lucene/codecs/lucene94/OffHeapVectorValues.java | 10 ----------
.../org/apache/lucene/index/BufferingKnnVectorsWriter.java | 5 -----
.../java/org/apache/lucene/index/FilterVectorValues.java | 5 -----
.../src/java/org/apache/lucene/index/VectorValues.java | 14 ++++++--------
.../test/org/apache/lucene/util/hnsw/MockVectorValues.java | 5 -----
.../test/org/apache/lucene/util/hnsw/TestHnswGraph.java | 5 -----
12 files changed, 9 insertions(+), 68 deletions(-)
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 7ad2f96e76c..2b5e0aeee81 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -34,6 +34,9 @@ API Changes
necessary and also illegal as it reported a number of dimensions equal to
zero. (Adrien Grand)
+* GITHUB#11962: VectorValues#cost() now delegates to VectorValues#size().
+ (Adrien Grand)
+
New Features
---------------------
* GITHUB#11795: Add ByteWritesTrackingDirectoryWrapper to expose metrics for bytes merged, flushed, and overall
diff --git a/lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene90/Lucene90HnswVectorsReader.java b/lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene90/Lucene90HnswVectorsReader.java
index 12cca5af7d1..83a3b090bdb 100644
--- a/lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene90/Lucene90HnswVectorsReader.java
+++ b/lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene90/Lucene90HnswVectorsReader.java
@@ -443,11 +443,6 @@ public final class Lucene90HnswVectorsReader extends KnnVectorsReader {
return doc;
}
- @Override
- public long cost() {
- return ordToDoc.length;
- }
-
@Override
public RandomAccessVectorValues copy() {
return new OffHeapVectorValues(dimension, ordToDoc, dataIn.clone());
diff --git a/lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene91/Lucene91HnswVectorsReader.java b/lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene91/Lucene91HnswVectorsReader.java
index 3eaac9e5ecb..d889ff4fbda 100644
--- a/lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene91/Lucene91HnswVectorsReader.java
+++ b/lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene91/Lucene91HnswVectorsReader.java
@@ -495,11 +495,6 @@ public final class Lucene91HnswVectorsReader extends KnnVectorsReader {
return doc;
}
- @Override
- public long cost() {
- return size;
- }
-
@Override
public RandomAccessVectorValues copy() {
return new OffHeapVectorValues(dimension, size, ordToDoc, dataIn.clone());
diff --git a/lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene92/OffHeapVectorValues.java b/lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene92/OffHeapVectorValues.java
index 0f913c45b7a..88968e42238 100644
--- a/lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene92/OffHeapVectorValues.java
+++ b/lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene92/OffHeapVectorValues.java
@@ -59,11 +59,6 @@ abstract class OffHeapVectorValues extends VectorValues implements RandomAccessV
return size;
}
- @Override
- public long cost() {
- return size;
- }
-
@Override
public float[] vectorValue(int targetOrd) throws IOException {
slice.seek((long) targetOrd * byteSize);
@@ -286,11 +281,6 @@ abstract class OffHeapVectorValues extends VectorValues implements RandomAccessV
return doc = NO_MORE_DOCS;
}
- @Override
- public long cost() {
- return 0;
- }
-
@Override
public RandomAccessVectorValues copy() throws IOException {
throw new UnsupportedOperationException();
diff --git a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java
index ed47670b4c6..4b29ce942d3 100644
--- a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java
+++ b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java
@@ -333,11 +333,6 @@ public class SimpleTextKnnVectorsReader extends KnnVectorsReader {
return slowAdvance(target);
}
- @Override
- public long cost() {
- return size();
- }
-
private void readAllVectors() throws IOException {
for (float[] value : values) {
readVector(value);
diff --git a/lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsWriter.java b/lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsWriter.java
index e15c6d8bc07..e45057ef37d 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsWriter.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsWriter.java
@@ -179,11 +179,6 @@ public abstract class KnnVectorsWriter implements Accountable, Closeable {
return size;
}
- @Override
- public long cost() {
- return size;
- }
-
@Override
public int dimension() {
return subs.get(0).values.dimension();
diff --git a/lucene/core/src/java/org/apache/lucene/codecs/lucene94/OffHeapVectorValues.java b/lucene/core/src/java/org/apache/lucene/codecs/lucene94/OffHeapVectorValues.java
index f2e40c6566a..c3dd4781e63 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene94/OffHeapVectorValues.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene94/OffHeapVectorValues.java
@@ -59,11 +59,6 @@ abstract class OffHeapVectorValues extends VectorValues implements RandomAccessV
return size;
}
- @Override
- public long cost() {
- return size;
- }
-
@Override
public float[] vectorValue(int targetOrd) throws IOException {
slice.seek((long) targetOrd * byteSize);
@@ -291,11 +286,6 @@ abstract class OffHeapVectorValues extends VectorValues implements RandomAccessV
return doc = NO_MORE_DOCS;
}
- @Override
- public long cost() {
- return 0;
- }
-
@Override
public RandomAccessVectorValues copy() throws IOException {
throw new UnsupportedOperationException();
diff --git a/lucene/core/src/java/org/apache/lucene/index/BufferingKnnVectorsWriter.java b/lucene/core/src/java/org/apache/lucene/index/BufferingKnnVectorsWriter.java
index e6283992a10..8b70ebcfd30 100644
--- a/lucene/core/src/java/org/apache/lucene/index/BufferingKnnVectorsWriter.java
+++ b/lucene/core/src/java/org/apache/lucene/index/BufferingKnnVectorsWriter.java
@@ -286,10 +286,5 @@ public abstract class BufferingKnnVectorsWriter extends KnnVectorsWriter {
public int advance(int target) {
throw new UnsupportedOperationException();
}
-
- @Override
- public long cost() {
- return docsWithFieldIter.cost();
- }
}
}
diff --git a/lucene/core/src/java/org/apache/lucene/index/FilterVectorValues.java b/lucene/core/src/java/org/apache/lucene/index/FilterVectorValues.java
index 32dfc37a0f1..6ab47be9753 100644
--- a/lucene/core/src/java/org/apache/lucene/index/FilterVectorValues.java
+++ b/lucene/core/src/java/org/apache/lucene/index/FilterVectorValues.java
@@ -48,11 +48,6 @@ public abstract class FilterVectorValues extends VectorValues {
return in.advance(target);
}
- @Override
- public long cost() {
- return in.cost();
- }
-
@Override
public int dimension() {
return in.dimension();
diff --git a/lucene/core/src/java/org/apache/lucene/index/VectorValues.java b/lucene/core/src/java/org/apache/lucene/index/VectorValues.java
index 2e1357567c9..21945e888e9 100644
--- a/lucene/core/src/java/org/apache/lucene/index/VectorValues.java
+++ b/lucene/core/src/java/org/apache/lucene/index/VectorValues.java
@@ -39,14 +39,17 @@ public abstract class VectorValues extends DocIdSetIterator {
public abstract int dimension();
/**
- * TODO: should we use cost() for this? We rely on its always being exactly the number of
- * documents having a value for this field, which is not guaranteed by the cost() contract, but in
- * all the implementations so far they are the same.
+ * Return the number of vectors for this field.
*
* @return the number of vectors returned by this iterator
*/
public abstract int size();
+ @Override
+ public final long cost() {
+ return size();
+ }
+
/**
* Return the vector value for the current document ID. It is illegal to call this method when the
* iterator is not positioned: before advancing, or after failing to advance. The returned array
@@ -127,10 +130,5 @@ public abstract class VectorValues extends DocIdSetIterator {
public int advance(int target) throws IOException {
throw new UnsupportedOperationException();
}
-
- @Override
- public long cost() {
- return size();
- }
}
}
diff --git a/lucene/core/src/test/org/apache/lucene/util/hnsw/MockVectorValues.java b/lucene/core/src/test/org/apache/lucene/util/hnsw/MockVectorValues.java
index 99404df4d11..2ae264fe8ce 100644
--- a/lucene/core/src/test/org/apache/lucene/util/hnsw/MockVectorValues.java
+++ b/lucene/core/src/test/org/apache/lucene/util/hnsw/MockVectorValues.java
@@ -122,9 +122,4 @@ class MockVectorValues extends VectorValues implements RandomAccessVectorValues
}
return NO_MORE_DOCS;
}
-
- @Override
- public long cost() {
- return size();
- }
}
diff --git a/lucene/core/src/test/org/apache/lucene/util/hnsw/TestHnswGraph.java b/lucene/core/src/test/org/apache/lucene/util/hnsw/TestHnswGraph.java
index 7cb192b9367..c3465f0bc68 100644
--- a/lucene/core/src/test/org/apache/lucene/util/hnsw/TestHnswGraph.java
+++ b/lucene/core/src/test/org/apache/lucene/util/hnsw/TestHnswGraph.java
@@ -767,11 +767,6 @@ public class TestHnswGraph extends LuceneTestCase {
return doc;
}
- @Override
- public long cost() {
- return size;
- }
-
@Override
public float[] vectorValue(int ord) {
return unitVector2d(ord / (double) size, value);
[lucene] 01/03: Remove VectorValues#EMPTY. (#11961)
Posted by jp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jpountz pushed a commit to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 604d320326ef5aac81acf38c942f2c1ccdf12ec8
Author: Adrien Grand <jp...@gmail.com>
AuthorDate: Wed Nov 23 10:52:12 2022 +0100
Remove VectorValues#EMPTY. (#11961)
This instance is illegal as it reports a number of dimensions equal to zero.
---
lucene/CHANGES.txt | 4 ++
.../simpletext/SimpleTextKnnVectorsReader.java | 3 +-
.../org/apache/lucene/codecs/KnnVectorsFormat.java | 5 +--
.../java/org/apache/lucene/index/VectorValues.java | 44 ----------------------
4 files changed, 8 insertions(+), 48 deletions(-)
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 1ac5177b026..7ad2f96e76c 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -30,6 +30,10 @@ API Changes
to avoid possible exceptions when building queries from an empty term list. The helper
TermAndBoost class now holds a BytesRef rather than a Term. (Alan Woodward)
+* GITHUB#11961: VectorValues#EMPTY was removed as this instance was not
+ necessary and also illegal as it reported a number of dimensions equal to
+ zero. (Adrien Grand)
+
New Features
---------------------
* GITHUB#11795: Add ByteWritesTrackingDirectoryWrapper to expose metrics for bytes merged, flushed, and overall
diff --git a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java
index 060f3decb75..ed47670b4c6 100644
--- a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java
+++ b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java
@@ -120,7 +120,8 @@ public class SimpleTextKnnVectorsReader extends KnnVectorsReader {
}
int dimension = info.getVectorDimension();
if (dimension == 0) {
- return VectorValues.EMPTY;
+ throw new IllegalStateException(
+ "KNN vectors readers should not be called on fields that don't enable KNN vectors");
}
FieldEntry fieldEntry = fieldEntries.get(field);
if (fieldEntry == null) {
diff --git a/lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsFormat.java b/lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsFormat.java
index 945b213b034..2fa4628aa77 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsFormat.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsFormat.java
@@ -22,7 +22,6 @@ import org.apache.lucene.index.SegmentReadState;
import org.apache.lucene.index.SegmentWriteState;
import org.apache.lucene.index.VectorValues;
import org.apache.lucene.search.TopDocs;
-import org.apache.lucene.search.TopDocsCollector;
import org.apache.lucene.util.Bits;
import org.apache.lucene.util.NamedSPILoader;
@@ -95,13 +94,13 @@ public abstract class KnnVectorsFormat implements NamedSPILoader.NamedSPI {
@Override
public VectorValues getVectorValues(String field) {
- return VectorValues.EMPTY;
+ throw new UnsupportedOperationException();
}
@Override
public TopDocs search(
String field, float[] target, int k, Bits acceptDocs, int visitedLimit) {
- return TopDocsCollector.EMPTY_TOPDOCS;
+ throw new UnsupportedOperationException();
}
@Override
diff --git a/lucene/core/src/java/org/apache/lucene/index/VectorValues.java b/lucene/core/src/java/org/apache/lucene/index/VectorValues.java
index f96185cd1ab..2e1357567c9 100644
--- a/lucene/core/src/java/org/apache/lucene/index/VectorValues.java
+++ b/lucene/core/src/java/org/apache/lucene/index/VectorValues.java
@@ -68,50 +68,6 @@ public abstract class VectorValues extends DocIdSetIterator {
throw new UnsupportedOperationException();
}
- /**
- * Represents the lack of vector values. It is returned by providers that do not support
- * VectorValues.
- */
- public static final VectorValues EMPTY =
- new VectorValues() {
-
- @Override
- public int size() {
- return 0;
- }
-
- @Override
- public int dimension() {
- return 0;
- }
-
- @Override
- public float[] vectorValue() {
- throw new IllegalStateException(
- "Attempt to get vectors from EMPTY values (which was not advanced)");
- }
-
- @Override
- public int docID() {
- throw new IllegalStateException("VectorValues is EMPTY, and not positioned on a doc");
- }
-
- @Override
- public int nextDoc() {
- return NO_MORE_DOCS;
- }
-
- @Override
- public int advance(int target) {
- return NO_MORE_DOCS;
- }
-
- @Override
- public long cost() {
- return 0;
- }
- };
-
/** Sorting VectorValues that iterate over documents in the order of the provided sortMap */
public static class SortingVectorValues extends VectorValues {
private final RandomAccessVectorValues randomAccess;
[lucene] 02/03: No longer announce releases on general@l.a.o. (#11967)
Posted by jp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jpountz pushed a commit to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit f17df395ab3138de11623f0048b680b053deb744
Author: Adrien Grand <jp...@gmail.com>
AuthorDate: Wed Nov 23 10:55:58 2022 +0100
No longer announce releases on general@l.a.o. (#11967)
This mailing-list is deprecated.
---
dev-tools/scripts/releaseWizard.yaml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/dev-tools/scripts/releaseWizard.yaml b/dev-tools/scripts/releaseWizard.yaml
index 51ee82d0085..7f793d30d2a 100644
--- a/dev-tools/scripts/releaseWizard.yaml
+++ b/dev-tools/scripts/releaseWizard.yaml
@@ -108,7 +108,7 @@ templates:
.Mail template
----
- To: dev@lucene.apache.org, general@lucene.apache.org, java-user@lucene.apache.org
+ To: dev@lucene.apache.org, java-user@lucene.apache.org
Subject: [ANNOUNCE] Apache Lucene {{ release_version }} released
(( template=announce_lucene_mail_body ))