You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ka...@apache.org on 2026/09/18 19:20:12 UTC
(lucene) branch branch_10_5 updated: Backwards-compatibility changes for [Byte|Float]VectorSimilarityQuery (#16671) (backport #16654)
This is an automated email from the ASF dual-hosted git repository.
kaivalnp pushed a commit to branch branch_10_5
in repository https://gitbox.apache.org/repos/asf/lucene.git
The following commit(s) were added to refs/heads/branch_10_5 by this push:
new 5caef2a1fda Backwards-compatibility changes for [Byte|Float]VectorSimilarityQuery (#16671) (backport #16654)
5caef2a1fda is described below
commit 5caef2a1fda9ea176f6e6b87501c6e8cdb5c1c05
Author: Kaival Parikh <ka...@gmail.com>
AuthorDate: Fri Sep 18 15:13:17 2026 -0400
Backwards-compatibility changes for [Byte|Float]VectorSimilarityQuery (#16671) (backport #16654)
- Lucene 10.5.0 (#15784) switched `[Byte|Float]VectorSimilarityQuery` from explicit to adaptive HNSW graph traversal but kept the constructor signatures unchanged, so the two positional float arguments silently changed meaning from `(traversalSimilarity, resultSimilarity)` to `(resultSimilarity, decay)`.
- Make the public constructors `private` to turn the silent behavior change into a compilation error on upgrade to 10.5.2
- Both behaviors are now exposed as nested classes: `[Byte|Float]VectorSimilarityQuery.Explicit` exposes the Lucene 10.4 behavior and is deprecated (will be removed in a future release), and `[Byte|Float]VectorSimilarityQuery.Adaptive` exposes the 10.5 behavior.
(cherry picked from commit bc9adc96cc58d1e20761d961a2ab865eaede3962)
---
lucene/CHANGES.txt | 11 +
lucene/MIGRATE.md | 3 +
.../lucene/search/ByteVectorSimilarityQuery.java | 264 +++++---
.../search/ExplicitVectorSimilarityCollector.java | 86 +++
.../lucene/search/FloatVectorSimilarityQuery.java | 264 +++++---
.../BaseExplicitVectorSimilarityQueryTestCase.java | 698 +++++++++++++++++++++
.../search/TestByteVectorSimilarityQuery.java | 6 +-
... => TestExplicitByteVectorSimilarityQuery.java} | 30 +-
...=> TestExplicitFloatVectorSimilarityQuery.java} | 30 +-
.../search/TestFloatVectorSimilarityQuery.java | 6 +-
.../search/TestVectorSimilarityCollector.java | 35 ++
11 files changed, 1249 insertions(+), 184 deletions(-)
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index dca44bfac06..1ed2ebab506 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -5,6 +5,17 @@ http://s.apache.org/luceneversions
======================= Lucene 10.5.2 =======================
+API Changes
+---------------------
+
+* GITHUB#16654: Public constructors of [Byte|Float]VectorSimilarityQuery are now private (binary-incompatible change).
+ Lucene 10.5.0 (GITHUB#15784) switched these queries from explicit to adaptive HNSW graph traversal but kept the
+ constructor signatures, so the two positional float arguments silently changed meaning from (traversalSimilarity,
+ resultSimilarity) to (resultSimilarity, decay). This turns that silent behavior change into a compilation error on
+ upgrade. Both behaviors are now exposed as nested classes: [Byte|Float]VectorSimilarityQuery.Explicit exposes the
+ Lucene 10.4 behavior and is deprecated, and [Byte|Float]VectorSimilarityQuery.Adaptive exposes the 10.5 behavior.
+ (Kaival Parikh, Chris Hostetter, Michael Sokolov)
+
Bug Fixes
---------------------
* GITHUB#16553: Fix stored-fields corruption when PerField#finish throws: the stored-fields frame
diff --git a/lucene/MIGRATE.md b/lucene/MIGRATE.md
index 80063790114..697d3779b95 100644
--- a/lucene/MIGRATE.md
+++ b/lucene/MIGRATE.md
@@ -38,6 +38,9 @@ values producing better recall using more graph exploration. This gives a better
in most cases, while still providing a knob for advanced users to tune quality and performance if needed (using the
`decay` factor).
+For Lucene 10.4 behavior, use `[Byte|Float]VectorSimilarityQuery.Explicit` (deprecated, and will be removed in a future
+release). For Lucene 10.5 behavior, use `[Byte|Float]VectorSimilarityQuery.Adaptive`.
+
## Migration from Lucene 9.x to Lucene 10.0
### DataInput#readVLong() may now read negative vlongs
diff --git a/lucene/core/src/java/org/apache/lucene/search/ByteVectorSimilarityQuery.java b/lucene/core/src/java/org/apache/lucene/search/ByteVectorSimilarityQuery.java
index 32cb765ac01..0e167e7c230 100644
--- a/lucene/core/src/java/org/apache/lucene/search/ByteVectorSimilarityQuery.java
+++ b/lucene/core/src/java/org/apache/lucene/search/ByteVectorSimilarityQuery.java
@@ -32,27 +32,202 @@ import org.apache.lucene.search.knn.KnnSearchStrategy.Hnsw;
*
* @lucene.experimental
*/
-public class ByteVectorSimilarityQuery extends AbstractVectorSimilarityQuery {
+public abstract sealed class ByteVectorSimilarityQuery extends AbstractVectorSimilarityQuery {
private final byte[] target;
+ /** A {@link ByteVectorSimilarityQuery} with an adaptive threshold for graph traversal. */
+ public static non-sealed class Adaptive extends ByteVectorSimilarityQuery {
+ /**
+ * Search for all (approximate) byte vectors above a similarity threshold using {@link
+ * VectorSimilarityCollector}, with a caller-supplied {@link KnnSearchStrategy}. If a filter is
+ * applied, it traverses as many nodes as the cost of the filter, and then falls back to exact
+ * search if results are incomplete.
+ *
+ * @param field a field that has been indexed as a {@link KnnByteVectorField}.
+ * @param target the target of the search.
+ * @param resultSimilarity similarity score for result collection.
+ * @param decay decay factor for graph traversal buffer.
+ * @param filter a filter applied before the vector search.
+ * @param searchStrategy the {@link KnnSearchStrategy} to use during graph search. If {@code
+ * null}, this query's own default is used: an {@link Hnsw} with {@code
+ * filteredSearchThreshold == 0}, which preserves this query's filter handling. Note this
+ * differs from {@link Hnsw#DEFAULT}, which uses a threshold of 60. The underlying format
+ * may not support all strategies and is free to ignore the requested strategy.
+ */
+ public Adaptive(
+ String field,
+ byte[] target,
+ float resultSimilarity,
+ float decay,
+ Query filter,
+ KnnSearchStrategy searchStrategy) {
+ super(field, target, resultSimilarity, decay, filter, searchStrategy);
+ }
+
+ /**
+ * Search for all (approximate) byte vectors above a similarity threshold using {@link
+ * VectorSimilarityCollector}, with the default {@link KnnSearchStrategy}. If a filter is
+ * applied, it traverses as many nodes as the cost of the filter, and then falls back to exact
+ * search if results are incomplete.
+ *
+ * @param field a field that has been indexed as a {@link KnnByteVectorField}.
+ * @param target the target of the search.
+ * @param resultSimilarity similarity score for result collection.
+ * @param decay decay factor for graph traversal buffer.
+ * @param filter a filter applied before the vector search.
+ */
+ public Adaptive(
+ String field, byte[] target, float resultSimilarity, float decay, Query filter) {
+ this(field, target, resultSimilarity, decay, filter, DEFAULT_STRATEGY);
+ }
+
+ /**
+ * Search for all (approximate) byte vectors above a similarity threshold using {@link
+ * VectorSimilarityCollector}. If a filter is applied, it traverses as many nodes as the cost of
+ * the filter, and then falls back to exact search if results are incomplete.
+ *
+ * @param field a field that has been indexed as a {@link KnnByteVectorField}.
+ * @param target the target of the search.
+ * @param resultSimilarity similarity score for result collection.
+ * @param filter a filter applied before the vector search.
+ */
+ public Adaptive(String field, byte[] target, float resultSimilarity, Query filter) {
+ this(field, target, resultSimilarity, DEFAULT_DECAY, filter);
+ }
+
+ /**
+ * Search for all (approximate) byte vectors above a similarity threshold using {@link
+ * VectorSimilarityCollector}.
+ *
+ * @param field a field that has been indexed as a {@link KnnByteVectorField}.
+ * @param target the target of the search.
+ * @param resultSimilarity similarity score for result collection.
+ */
+ public Adaptive(String field, byte[] target, float resultSimilarity) {
+ this(field, target, resultSimilarity, null);
+ }
+
+ @Override
+ public String toString(String field) {
+ return String.format(
+ Locale.ROOT,
+ "ByteVectorSimilarityQuery.Adaptive[field=%s target=[%d...] resultSimilarity=%f decay=%f filter=%s]",
+ field,
+ super.target[0],
+ resultSimilarity,
+ decay,
+ filter);
+ }
+ }
+
/**
- * Search for all (approximate) byte vectors above a similarity threshold using {@link
- * VectorSimilarityCollector}, with a caller-supplied {@link KnnSearchStrategy}. If a filter is
- * applied, it traverses as many nodes as the cost of the filter, and then falls back to exact
- * search if results are incomplete.
+ * A {@link ByteVectorSimilarityQuery} with an explicit threshold for graph traversal.
*
- * @param field a field that has been indexed as a {@link KnnByteVectorField}.
- * @param target the target of the search.
- * @param resultSimilarity similarity score for result collection.
- * @param decay decay factor for graph traversal buffer.
- * @param filter a filter applied before the vector search.
- * @param searchStrategy the {@link KnnSearchStrategy} to use during graph search. If {@code
- * null}, this query's own default is used: an {@link Hnsw} with {@code
- * filteredSearchThreshold == 0}, which preserves this query's filter handling. Note this
- * differs from {@link Hnsw#DEFAULT}, which uses a threshold of 60. The underlying format may
- * not support all strategies and is free to ignore the requested strategy.
+ * @deprecated Equivalent to the {@link ByteVectorSimilarityQuery} constructors in Lucene 10.4 and
+ * earlier, use {@link Adaptive} for a more performant version.
*/
- public ByteVectorSimilarityQuery(
+ @Deprecated
+ public static non-sealed class Explicit extends ByteVectorSimilarityQuery {
+ private final float traversalSimilarity;
+
+ /**
+ * Search for all (approximate) byte vectors above a similarity threshold using {@link
+ * ExplicitVectorSimilarityCollector}. If a filter is applied, it traverses as many nodes as the
+ * cost of the filter, and then falls back to exact search if results are incomplete.
+ *
+ * @param field a field that has been indexed as a {@link KnnByteVectorField}.
+ * @param target the target of the search.
+ * @param traversalSimilarity (lower) similarity score for graph traversal.
+ * @param resultSimilarity (higher) similarity score for result collection.
+ * @param filter a filter applied before the vector search.
+ */
+ public Explicit(
+ String field,
+ byte[] target,
+ float traversalSimilarity,
+ float resultSimilarity,
+ Query filter) {
+ super(field, target, resultSimilarity, 0f, filter, DEFAULT_STRATEGY);
+ if (traversalSimilarity > resultSimilarity) {
+ throw new IllegalArgumentException("traversalSimilarity should be <= resultSimilarity");
+ }
+ this.traversalSimilarity = traversalSimilarity;
+ }
+
+ /**
+ * Search for all (approximate) byte vectors above a similarity threshold using {@link
+ * ExplicitVectorSimilarityCollector}.
+ *
+ * @param field a field that has been indexed as a {@link KnnByteVectorField}.
+ * @param target the target of the search.
+ * @param traversalSimilarity (lower) similarity score for graph traversal.
+ * @param resultSimilarity (higher) similarity score for result collection.
+ */
+ public Explicit(
+ String field, byte[] target, float traversalSimilarity, float resultSimilarity) {
+ this(field, target, traversalSimilarity, resultSimilarity, null);
+ }
+
+ /**
+ * Search for all (approximate) byte vectors above a similarity threshold using {@link
+ * ExplicitVectorSimilarityCollector}. If a filter is applied, it traverses as many nodes as the
+ * cost of the filter, and then falls back to exact search if results are incomplete.
+ *
+ * @param field a field that has been indexed as a {@link KnnByteVectorField}.
+ * @param target the target of the search.
+ * @param resultSimilarity similarity score for result collection.
+ * @param filter a filter applied before the vector search.
+ */
+ public Explicit(String field, byte[] target, float resultSimilarity, Query filter) {
+ this(field, target, resultSimilarity, resultSimilarity, filter);
+ }
+
+ /**
+ * Search for all (approximate) byte vectors above a similarity threshold using {@link
+ * ExplicitVectorSimilarityCollector}.
+ *
+ * @param field a field that has been indexed as a {@link KnnByteVectorField}.
+ * @param target the target of the search.
+ * @param resultSimilarity similarity score for result collection.
+ */
+ public Explicit(String field, byte[] target, float resultSimilarity) {
+ this(field, target, resultSimilarity, resultSimilarity, null);
+ }
+
+ @Override
+ protected KnnCollectorManager getKnnCollectorManager() {
+ return (visitedLimit, strategy, context) ->
+ new ExplicitVectorSimilarityCollector(
+ traversalSimilarity, resultSimilarity, visitedLimit);
+ }
+
+ @Override
+ public String toString(String field) {
+ return String.format(
+ Locale.ROOT,
+ "ByteVectorSimilarityQuery.Explicit[field=%s target=[%d...] traversalSimilarity=%f resultSimilarity=%f filter=%s]",
+ field,
+ super.target[0],
+ traversalSimilarity,
+ resultSimilarity,
+ filter);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ return super.equals(o)
+ && Float.compare(traversalSimilarity, ((Explicit) o).traversalSimilarity) == 0;
+ }
+
+ @Override
+ public int hashCode() {
+ int result = super.hashCode();
+ result = 31 * result + Float.hashCode(traversalSimilarity);
+ return result;
+ }
+ }
+
+ private ByteVectorSimilarityQuery(
String field,
byte[] target,
float resultSimilarity,
@@ -63,50 +238,6 @@ public class ByteVectorSimilarityQuery extends AbstractVectorSimilarityQuery {
this.target = Objects.requireNonNull(target, "target");
}
- /**
- * Search for all (approximate) byte vectors above a similarity threshold using {@link
- * VectorSimilarityCollector}, with the default {@link KnnSearchStrategy}. If a filter is applied,
- * it traverses as many nodes as the cost of the filter, and then falls back to exact search if
- * results are incomplete.
- *
- * @param field a field that has been indexed as a {@link KnnByteVectorField}.
- * @param target the target of the search.
- * @param decay decay factor for graph traversal buffer.
- * @param resultSimilarity similarity score for result collection.
- * @param filter a filter applied before the vector search.
- */
- public ByteVectorSimilarityQuery(
- String field, byte[] target, float resultSimilarity, float decay, Query filter) {
- this(field, target, resultSimilarity, decay, filter, DEFAULT_STRATEGY);
- }
-
- /**
- * Search for all (approximate) byte vectors above a similarity threshold using {@link
- * VectorSimilarityCollector}. If a filter is applied, it traverses as many nodes as the cost of
- * the filter, and then falls back to exact search if results are incomplete.
- *
- * @param field a field that has been indexed as a {@link KnnByteVectorField}.
- * @param target the target of the search.
- * @param resultSimilarity similarity score for result collection.
- * @param filter a filter applied before the vector search.
- */
- public ByteVectorSimilarityQuery(
- String field, byte[] target, float resultSimilarity, Query filter) {
- this(field, target, resultSimilarity, DEFAULT_DECAY, filter);
- }
-
- /**
- * Search for all (approximate) byte vectors above a similarity threshold using {@link
- * VectorSimilarityCollector}.
- *
- * @param field a field that has been indexed as a {@link KnnByteVectorField}.
- * @param target the target of the search.
- * @param resultSimilarity similarity score for result collection.
- */
- public ByteVectorSimilarityQuery(String field, byte[] target, float resultSimilarity) {
- this(field, target, resultSimilarity, null);
- }
-
@Override
VectorScorer createVectorScorer(LeafReaderContext context) throws IOException {
ByteVectorValues vectorValues = context.reader().getByteVectorValues(field);
@@ -130,19 +261,6 @@ public class ByteVectorSimilarityQuery extends AbstractVectorSimilarityQuery {
return collector.topDocs();
}
- @Override
- public String toString(String field) {
- return String.format(
- Locale.ROOT,
- "%s[field=%s target=[%d...] resultSimilarity=%f decay=%f filter=%s]",
- getClass().getSimpleName(),
- field,
- target[0],
- resultSimilarity,
- decay,
- filter);
- }
-
@Override
public boolean equals(Object o) {
return sameClassAs(o)
diff --git a/lucene/core/src/java/org/apache/lucene/search/ExplicitVectorSimilarityCollector.java b/lucene/core/src/java/org/apache/lucene/search/ExplicitVectorSimilarityCollector.java
new file mode 100644
index 00000000000..72eec5e231b
--- /dev/null
+++ b/lucene/core/src/java/org/apache/lucene/search/ExplicitVectorSimilarityCollector.java
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.search;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Perform a similarity-based graph search.
+ *
+ * @deprecated Use {@link VectorSimilarityCollector} for a more performant version.
+ * @lucene.experimental
+ */
+@Deprecated
+class ExplicitVectorSimilarityCollector extends AbstractKnnCollector {
+ private final float traversalSimilarity, resultSimilarity;
+ private float maxSimilarity;
+ private final List<ScoreDoc> scoreDocList;
+
+ /**
+ * Perform a similarity-based graph search. The graph is traversed till better scoring nodes are
+ * available, or the best candidate is below {@link #traversalSimilarity}. All traversed nodes
+ * above {@link #resultSimilarity} are collected.
+ *
+ * @param traversalSimilarity (lower) similarity score for graph traversal.
+ * @param resultSimilarity (higher) similarity score for result collection.
+ * @param visitLimit limit on number of nodes to visit.
+ */
+ public ExplicitVectorSimilarityCollector(
+ float traversalSimilarity, float resultSimilarity, long visitLimit) {
+ // TODO: add search strategy support
+ super(1, visitLimit, AbstractVectorSimilarityQuery.DEFAULT_STRATEGY);
+ if (traversalSimilarity > resultSimilarity) {
+ throw new IllegalArgumentException("traversalSimilarity should be <= resultSimilarity");
+ }
+ this.traversalSimilarity = traversalSimilarity;
+ this.resultSimilarity = resultSimilarity;
+ this.maxSimilarity = Float.NEGATIVE_INFINITY;
+ this.scoreDocList = new ArrayList<>();
+ }
+
+ @Override
+ public boolean collect(int docId, float similarity) {
+ maxSimilarity = Math.max(maxSimilarity, similarity);
+ if (similarity >= resultSimilarity) {
+ scoreDocList.add(new ScoreDoc(docId, similarity));
+ }
+ return true;
+ }
+
+ @Override
+ public float minCompetitiveSimilarity() {
+ return Math.min(traversalSimilarity, maxSimilarity);
+ }
+
+ @Override
+ public TopDocs topDocs() {
+ // Results are not returned in a sorted order to prevent unnecessary calculations (because we do
+ // not need to maintain the topK)
+ TotalHits.Relation relation =
+ earlyTerminated()
+ ? TotalHits.Relation.GREATER_THAN_OR_EQUAL_TO
+ : TotalHits.Relation.EQUAL_TO;
+ return new TopDocs(
+ new TotalHits(visitedCount(), relation), scoreDocList.toArray(ScoreDoc[]::new));
+ }
+
+ @Override
+ public int numCollected() {
+ return scoreDocList.size();
+ }
+}
diff --git a/lucene/core/src/java/org/apache/lucene/search/FloatVectorSimilarityQuery.java b/lucene/core/src/java/org/apache/lucene/search/FloatVectorSimilarityQuery.java
index af443e4314b..0d7ddcd881e 100644
--- a/lucene/core/src/java/org/apache/lucene/search/FloatVectorSimilarityQuery.java
+++ b/lucene/core/src/java/org/apache/lucene/search/FloatVectorSimilarityQuery.java
@@ -33,27 +33,202 @@ import org.apache.lucene.util.VectorUtil;
*
* @lucene.experimental
*/
-public class FloatVectorSimilarityQuery extends AbstractVectorSimilarityQuery {
+public abstract sealed class FloatVectorSimilarityQuery extends AbstractVectorSimilarityQuery {
private final float[] target;
+ /** A {@link FloatVectorSimilarityQuery} with an adaptive threshold for graph traversal. */
+ public static non-sealed class Adaptive extends FloatVectorSimilarityQuery {
+ /**
+ * Search for all (approximate) float vectors above a similarity threshold using {@link
+ * VectorSimilarityCollector}, with a caller-supplied {@link KnnSearchStrategy}. If a filter is
+ * applied, it traverses as many nodes as the cost of the filter, and then falls back to exact
+ * search if results are incomplete.
+ *
+ * @param field a field that has been indexed as a {@link KnnFloatVectorField}.
+ * @param target the target of the search.
+ * @param resultSimilarity similarity score for result collection.
+ * @param decay decay factor for graph traversal buffer.
+ * @param filter a filter applied before the vector search.
+ * @param searchStrategy the {@link KnnSearchStrategy} to use during graph search. If {@code
+ * null}, this query's own default is used: an {@link Hnsw} with {@code
+ * filteredSearchThreshold == 0}, which preserves this query's filter handling. Note this
+ * differs from {@link Hnsw#DEFAULT}, which uses a threshold of 60. The underlying format
+ * may not support all strategies and is free to ignore the requested strategy.
+ */
+ public Adaptive(
+ String field,
+ float[] target,
+ float resultSimilarity,
+ float decay,
+ Query filter,
+ KnnSearchStrategy searchStrategy) {
+ super(field, target, resultSimilarity, decay, filter, searchStrategy);
+ }
+
+ /**
+ * Search for all (approximate) float vectors above a similarity threshold using {@link
+ * VectorSimilarityCollector}, with the default {@link KnnSearchStrategy}. If a filter is
+ * applied, it traverses as many nodes as the cost of the filter, and then falls back to exact
+ * search if results are incomplete.
+ *
+ * @param field a field that has been indexed as a {@link KnnFloatVectorField}.
+ * @param target the target of the search.
+ * @param resultSimilarity similarity score for result collection.
+ * @param decay decay factor for graph traversal buffer.
+ * @param filter a filter applied before the vector search.
+ */
+ public Adaptive(
+ String field, float[] target, float resultSimilarity, float decay, Query filter) {
+ this(field, target, resultSimilarity, decay, filter, DEFAULT_STRATEGY);
+ }
+
+ /**
+ * Search for all (approximate) float vectors above a similarity threshold using {@link
+ * VectorSimilarityCollector}. If a filter is applied, it traverses as many nodes as the cost of
+ * the filter, and then falls back to exact search if results are incomplete.
+ *
+ * @param field a field that has been indexed as a {@link KnnFloatVectorField}.
+ * @param target the target of the search.
+ * @param resultSimilarity similarity score for result collection.
+ * @param filter a filter applied before the vector search.
+ */
+ public Adaptive(String field, float[] target, float resultSimilarity, Query filter) {
+ this(field, target, resultSimilarity, DEFAULT_DECAY, filter);
+ }
+
+ /**
+ * Search for all (approximate) float vectors above a similarity threshold using {@link
+ * VectorSimilarityCollector}.
+ *
+ * @param field a field that has been indexed as a {@link KnnFloatVectorField}.
+ * @param target the target of the search.
+ * @param resultSimilarity similarity score for result collection.
+ */
+ public Adaptive(String field, float[] target, float resultSimilarity) {
+ this(field, target, resultSimilarity, null);
+ }
+
+ @Override
+ public String toString(String field) {
+ return String.format(
+ Locale.ROOT,
+ "FloatVectorSimilarityQuery.Adaptive[field=%s target=[%f...] resultSimilarity=%f decay=%f filter=%s]",
+ field,
+ super.target[0],
+ resultSimilarity,
+ decay,
+ filter);
+ }
+ }
+
/**
- * Search for all (approximate) float vectors above a similarity threshold using {@link
- * VectorSimilarityCollector}, with a caller-supplied {@link KnnSearchStrategy}. If a filter is
- * applied, it traverses as many nodes as the cost of the filter, and then falls back to exact
- * search if results are incomplete.
+ * A {@link FloatVectorSimilarityQuery} with an explicit threshold for graph traversal.
*
- * @param field a field that has been indexed as a {@link KnnFloatVectorField}.
- * @param target the target of the search.
- * @param resultSimilarity similarity score for result collection.
- * @param decay decay factor for graph traversal buffer.
- * @param filter a filter applied before the vector search.
- * @param searchStrategy the {@link KnnSearchStrategy} to use during graph search. If {@code
- * null}, this query's own default is used: an {@link Hnsw} with {@code
- * filteredSearchThreshold == 0}, which preserves this query's filter handling. Note this
- * differs from {@link Hnsw#DEFAULT}, which uses a threshold of 60. The underlying format may
- * not support all strategies and is free to ignore the requested strategy.
+ * @deprecated Equivalent to the {@link FloatVectorSimilarityQuery} constructors in Lucene 10.4
+ * and earlier, use {@link Adaptive} for a more performant version.
*/
- public FloatVectorSimilarityQuery(
+ @Deprecated
+ public static non-sealed class Explicit extends FloatVectorSimilarityQuery {
+ private final float traversalSimilarity;
+
+ /**
+ * Search for all (approximate) float vectors above a similarity threshold using {@link
+ * ExplicitVectorSimilarityCollector}. If a filter is applied, it traverses as many nodes as the
+ * cost of the filter, and then falls back to exact search if results are incomplete.
+ *
+ * @param field a field that has been indexed as a {@link KnnFloatVectorField}.
+ * @param target the target of the search.
+ * @param traversalSimilarity (lower) similarity score for graph traversal.
+ * @param resultSimilarity (higher) similarity score for result collection.
+ * @param filter a filter applied before the vector search.
+ */
+ public Explicit(
+ String field,
+ float[] target,
+ float traversalSimilarity,
+ float resultSimilarity,
+ Query filter) {
+ super(field, target, resultSimilarity, 0f, filter, DEFAULT_STRATEGY);
+ if (traversalSimilarity > resultSimilarity) {
+ throw new IllegalArgumentException("traversalSimilarity should be <= resultSimilarity");
+ }
+ this.traversalSimilarity = traversalSimilarity;
+ }
+
+ /**
+ * Search for all (approximate) float vectors above a similarity threshold using {@link
+ * ExplicitVectorSimilarityCollector}.
+ *
+ * @param field a field that has been indexed as a {@link KnnFloatVectorField}.
+ * @param target the target of the search.
+ * @param traversalSimilarity (lower) similarity score for graph traversal.
+ * @param resultSimilarity (higher) similarity score for result collection.
+ */
+ public Explicit(
+ String field, float[] target, float traversalSimilarity, float resultSimilarity) {
+ this(field, target, traversalSimilarity, resultSimilarity, null);
+ }
+
+ /**
+ * Search for all (approximate) float vectors above a similarity threshold using {@link
+ * ExplicitVectorSimilarityCollector}. If a filter is applied, it traverses as many nodes as the
+ * cost of the filter, and then falls back to exact search if results are incomplete.
+ *
+ * @param field a field that has been indexed as a {@link KnnFloatVectorField}.
+ * @param target the target of the search.
+ * @param resultSimilarity similarity score for result collection.
+ * @param filter a filter applied before the vector search.
+ */
+ public Explicit(String field, float[] target, float resultSimilarity, Query filter) {
+ this(field, target, resultSimilarity, resultSimilarity, filter);
+ }
+
+ /**
+ * Search for all (approximate) float vectors above a similarity threshold using {@link
+ * ExplicitVectorSimilarityCollector}.
+ *
+ * @param field a field that has been indexed as a {@link KnnFloatVectorField}.
+ * @param target the target of the search.
+ * @param resultSimilarity similarity score for result collection.
+ */
+ public Explicit(String field, float[] target, float resultSimilarity) {
+ this(field, target, resultSimilarity, resultSimilarity, null);
+ }
+
+ @Override
+ protected KnnCollectorManager getKnnCollectorManager() {
+ return (visitedLimit, strategy, context) ->
+ new ExplicitVectorSimilarityCollector(
+ traversalSimilarity, resultSimilarity, visitedLimit);
+ }
+
+ @Override
+ public String toString(String field) {
+ return String.format(
+ Locale.ROOT,
+ "FloatVectorSimilarityQuery.Explicit[field=%s target=[%f...] traversalSimilarity=%f resultSimilarity=%f filter=%s]",
+ field,
+ super.target[0],
+ traversalSimilarity,
+ resultSimilarity,
+ filter);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ return super.equals(o)
+ && Float.compare(traversalSimilarity, ((Explicit) o).traversalSimilarity) == 0;
+ }
+
+ @Override
+ public int hashCode() {
+ int result = super.hashCode();
+ result = 31 * result + Float.hashCode(traversalSimilarity);
+ return result;
+ }
+ }
+
+ private FloatVectorSimilarityQuery(
String field,
float[] target,
float resultSimilarity,
@@ -64,50 +239,6 @@ public class FloatVectorSimilarityQuery extends AbstractVectorSimilarityQuery {
this.target = VectorUtil.checkFinite(Objects.requireNonNull(target, "target"));
}
- /**
- * Search for all (approximate) float vectors above a similarity threshold using {@link
- * VectorSimilarityCollector}, with the default {@link KnnSearchStrategy}. If a filter is applied,
- * it traverses as many nodes as the cost of the filter, and then falls back to exact search if
- * results are incomplete.
- *
- * @param field a field that has been indexed as a {@link KnnFloatVectorField}.
- * @param target the target of the search.
- * @param resultSimilarity similarity score for result collection.
- * @param decay decay factor for graph traversal buffer.
- * @param filter a filter applied before the vector search.
- */
- public FloatVectorSimilarityQuery(
- String field, float[] target, float resultSimilarity, float decay, Query filter) {
- this(field, target, resultSimilarity, decay, filter, DEFAULT_STRATEGY);
- }
-
- /**
- * Search for all (approximate) float vectors above a similarity threshold using {@link
- * VectorSimilarityCollector}. If a filter is applied, it traverses as many nodes as the cost of
- * the filter, and then falls back to exact search if results are incomplete.
- *
- * @param field a field that has been indexed as a {@link KnnFloatVectorField}.
- * @param target the target of the search.
- * @param resultSimilarity similarity score for result collection.
- * @param filter a filter applied before the vector search.
- */
- public FloatVectorSimilarityQuery(
- String field, float[] target, float resultSimilarity, Query filter) {
- this(field, target, resultSimilarity, DEFAULT_DECAY, filter);
- }
-
- /**
- * Search for all (approximate) float vectors above a similarity threshold using {@link
- * VectorSimilarityCollector}.
- *
- * @param field a field that has been indexed as a {@link KnnFloatVectorField}.
- * @param target the target of the search.
- * @param resultSimilarity similarity score for result collection.
- */
- public FloatVectorSimilarityQuery(String field, float[] target, float resultSimilarity) {
- this(field, target, resultSimilarity, null);
- }
-
@Override
VectorScorer createVectorScorer(LeafReaderContext context) throws IOException {
@SuppressWarnings("resource")
@@ -132,19 +263,6 @@ public class FloatVectorSimilarityQuery extends AbstractVectorSimilarityQuery {
return collector.topDocs();
}
- @Override
- public String toString(String field) {
- return String.format(
- Locale.ROOT,
- "%s[field=%s target=[%f...] resultSimilarity=%f decay=%f filter=%s]",
- getClass().getSimpleName(),
- field,
- target[0],
- resultSimilarity,
- decay,
- filter);
- }
-
@Override
public boolean equals(Object o) {
return sameClassAs(o)
diff --git a/lucene/core/src/test/org/apache/lucene/search/BaseExplicitVectorSimilarityQueryTestCase.java b/lucene/core/src/test/org/apache/lucene/search/BaseExplicitVectorSimilarityQueryTestCase.java
new file mode 100644
index 00000000000..0a5c627708c
--- /dev/null
+++ b/lucene/core/src/test/org/apache/lucene/search/BaseExplicitVectorSimilarityQueryTestCase.java
@@ -0,0 +1,698 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.search;
+
+import static org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsFormat.DEFAULT_BEAM_WIDTH;
+import static org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsFormat.DEFAULT_MAX_CONN;
+import static org.apache.lucene.search.DocIdSetIterator.NO_MORE_DOCS;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.IntStream;
+import org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsFormat;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.Field;
+import org.apache.lucene.document.IntField;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.QueryTimeout;
+import org.apache.lucene.index.Term;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.tests.index.RandomIndexWriter;
+import org.apache.lucene.tests.util.LuceneTestCase;
+import org.apache.lucene.tests.util.TestUtil;
+import org.apache.lucene.util.hnsw.HnswUtil;
+
+@Deprecated
+@LuceneTestCase.SuppressCodecs("SimpleText")
+abstract class BaseExplicitVectorSimilarityQueryTestCase<
+ V, F extends Field, Q extends AbstractVectorSimilarityQuery>
+ extends LuceneTestCase {
+ String vectorField, idField;
+ VectorSimilarityFunction function;
+ int numDocs, dim;
+
+ abstract V getRandomVector(int dim);
+
+ abstract float compare(V vector1, V vector2);
+
+ abstract boolean checkEquals(V vector1, V vector2);
+
+ abstract F getVectorField(String name, V vector, VectorSimilarityFunction function);
+
+ abstract Q getVectorQuery(
+ String field, V vector, float traversalSimilarity, float resultSimilarity, Query filter);
+
+ abstract Q getThrowingVectorQuery(
+ String field, V vector, float traversalSimilarity, float resultSimilarity, Query filter);
+
+ public void testEquals() {
+ String field1 = "f1", field2 = "f2";
+
+ V vector1 = getRandomVector(dim);
+ V vector2;
+ do {
+ vector2 = getRandomVector(dim);
+ } while (checkEquals(vector1, vector2));
+
+ float traversalSimilarity1 = 0.3f, traversalSimilarity2 = 0.4f;
+ float resultSimilarity1 = 0.4f, resultSimilarity2 = 0.5f;
+
+ Query filter1 = new TermQuery(new Term("t1", "v1"));
+ Query filter2 = new TermQuery(new Term("t2", "v2"));
+
+ Query query = getVectorQuery(field1, vector1, traversalSimilarity1, resultSimilarity1, filter1);
+
+ // Everything is equal
+ assertEquals(
+ query, getVectorQuery(field1, vector1, traversalSimilarity1, resultSimilarity1, filter1));
+
+ // Null check
+ assertNotEquals(query, null);
+
+ // Different field
+ assertNotEquals(
+ query, getVectorQuery(field2, vector1, traversalSimilarity1, resultSimilarity1, filter1));
+
+ // Different vector
+ assertNotEquals(
+ query, getVectorQuery(field1, vector2, traversalSimilarity1, resultSimilarity1, filter1));
+
+ // Different traversalSimilarity
+ assertNotEquals(
+ query, getVectorQuery(field1, vector1, traversalSimilarity2, resultSimilarity1, filter1));
+
+ // Different resultSimilarity
+ assertNotEquals(
+ query, getVectorQuery(field1, vector1, traversalSimilarity1, resultSimilarity2, filter1));
+
+ // Different filter
+ assertNotEquals(
+ query, getVectorQuery(field1, vector1, traversalSimilarity1, resultSimilarity1, filter2));
+ }
+
+ public void testEmptyIndex() throws IOException {
+ // Do not index any vectors
+ numDocs = 0;
+
+ try (Directory indexStore = getIndexStore(getRandomVectors(numDocs, dim));
+ IndexReader reader = DirectoryReader.open(indexStore)) {
+ IndexSearcher searcher = newSearcher(reader);
+
+ Query query =
+ getVectorQuery(
+ vectorField,
+ getRandomVector(dim),
+ Float.NEGATIVE_INFINITY,
+ Float.NEGATIVE_INFINITY,
+ null);
+
+ // Check that no vectors are found
+ assertEquals(0, searcher.count(query));
+ }
+ }
+
+ public void testExtremes() throws IOException {
+ try (Directory indexStore = getIndexStore(getRandomVectors(numDocs, dim));
+ IndexReader reader = DirectoryReader.open(indexStore)) {
+ IndexSearcher searcher = newSearcher(reader);
+ assumeTrue("graph is disconnected", HnswUtil.graphIsRooted(reader, vectorField));
+
+ // All vectors are above -Infinity
+ Query query1 =
+ getVectorQuery(
+ vectorField,
+ getRandomVector(dim),
+ Float.NEGATIVE_INFINITY,
+ Float.NEGATIVE_INFINITY,
+ null);
+
+ // Check that all vectors are found
+ assertEquals(numDocs, searcher.count(query1));
+
+ // No vectors are above +Infinity
+ Query query2 =
+ getVectorQuery(
+ vectorField,
+ getRandomVector(dim),
+ Float.POSITIVE_INFINITY,
+ Float.POSITIVE_INFINITY,
+ null);
+
+ // Check that no vectors are found
+ assertEquals(0, searcher.count(query2));
+ }
+ }
+
+ public void testRandomFilter() throws IOException {
+ // Filter a sub-range from 0 to numDocs
+ int startIndex = random().nextInt(numDocs);
+ int endIndex = random().nextInt(startIndex, numDocs);
+ Query filter = IntField.newRangeQuery(idField, startIndex, endIndex);
+
+ try (Directory indexStore = getIndexStore(getRandomVectors(numDocs, dim));
+ IndexReader reader = DirectoryReader.open(indexStore)) {
+ assumeTrue("graph is disconnected", HnswUtil.graphIsRooted(reader, vectorField));
+ IndexSearcher searcher = newSearcher(reader);
+
+ Query query =
+ getVectorQuery(
+ vectorField,
+ getRandomVector(dim),
+ Float.NEGATIVE_INFINITY,
+ Float.NEGATIVE_INFINITY,
+ filter);
+
+ ScoreDoc[] scoreDocs = searcher.search(query, numDocs).scoreDocs;
+ for (ScoreDoc scoreDoc : scoreDocs) {
+ int id = getId(searcher, scoreDoc.doc);
+
+ // Check that returned document is in selected range
+ assertTrue(id >= startIndex && id <= endIndex);
+ }
+ // Check that all filtered vectors are found
+ assertEquals(endIndex - startIndex + 1, scoreDocs.length);
+ }
+ }
+
+ public void testFilterWithNoMatches() throws IOException {
+ try (Directory indexStore = getIndexStore(getRandomVectors(numDocs, dim));
+ IndexReader reader = DirectoryReader.open(indexStore)) {
+ IndexSearcher searcher = newSearcher(reader);
+
+ // Non-existent field
+ Query filter1 = new TermQuery(new Term("random_field", "random_value"));
+ Query query1 =
+ getVectorQuery(
+ vectorField,
+ getRandomVector(dim),
+ Float.NEGATIVE_INFINITY,
+ Float.NEGATIVE_INFINITY,
+ filter1);
+
+ // Check that no vectors are found
+ assertEquals(0, searcher.count(query1));
+
+ // Field exists, but value of -1 is not indexed
+ Query filter2 = IntField.newExactQuery(idField, -1);
+ Query query2 =
+ getVectorQuery(
+ vectorField,
+ getRandomVector(dim),
+ Float.NEGATIVE_INFINITY,
+ Float.NEGATIVE_INFINITY,
+ filter2);
+
+ // Check that no vectors are found
+ assertEquals(0, searcher.count(query2));
+ }
+ }
+
+ public void testDimensionMismatch() throws IOException {
+ // Different dimension
+ int newDim = atLeast(dim + 1);
+
+ try (Directory indexStore = getIndexStore(getRandomVectors(numDocs, dim));
+ IndexReader reader = DirectoryReader.open(indexStore)) {
+ IndexSearcher searcher = newSearcher(reader);
+
+ Query query =
+ getVectorQuery(
+ vectorField,
+ getRandomVector(newDim),
+ Float.NEGATIVE_INFINITY,
+ Float.NEGATIVE_INFINITY,
+ null);
+
+ // Check that an exception for differing dimensions is thrown
+ IllegalArgumentException e =
+ expectThrows(IllegalArgumentException.class, () -> searcher.count(query));
+ assertEquals(
+ String.format(
+ Locale.ROOT,
+ "vector query dimension: %d differs from field dimension: %d",
+ newDim,
+ dim),
+ e.getMessage());
+ }
+ }
+
+ public void testNonVectorsField() throws IOException {
+ try (Directory indexStore = getIndexStore(getRandomVectors(numDocs, dim));
+ IndexReader reader = DirectoryReader.open(indexStore)) {
+ IndexSearcher searcher = newSearcher(reader);
+
+ // Non-existent field
+ Query query1 =
+ getVectorQuery(
+ "random_field",
+ getRandomVector(dim),
+ Float.NEGATIVE_INFINITY,
+ Float.NEGATIVE_INFINITY,
+ null);
+ assertEquals(0, searcher.count(query1));
+
+ // Indexed as int field
+ Query query2 =
+ getVectorQuery(
+ idField,
+ getRandomVector(dim),
+ Float.NEGATIVE_INFINITY,
+ Float.NEGATIVE_INFINITY,
+ null);
+ assertEquals(0, searcher.count(query2));
+ }
+ }
+
+ public void testSomeDeletes() throws IOException {
+ // Delete a sub-range from 0 to numDocs
+ int startIndex = random().nextInt(numDocs);
+ int endIndex = random().nextInt(startIndex, numDocs);
+ Query delete = IntField.newRangeQuery(idField, startIndex, endIndex);
+
+ try (Directory indexStore = getIndexStore(getRandomVectors(numDocs, dim));
+ IndexWriter w = new IndexWriter(indexStore, newIndexWriterConfig())) {
+
+ w.deleteDocuments(delete);
+ w.commit();
+
+ try (IndexReader reader = DirectoryReader.open(indexStore)) {
+ assumeTrue("graph is disconnected", HnswUtil.graphIsRooted(reader, vectorField));
+ IndexSearcher searcher = newSearcher(reader);
+
+ Query query =
+ getVectorQuery(
+ vectorField,
+ getRandomVector(dim),
+ Float.NEGATIVE_INFINITY,
+ Float.NEGATIVE_INFINITY,
+ null);
+
+ ScoreDoc[] scoreDocs = searcher.search(query, numDocs).scoreDocs;
+ for (ScoreDoc scoreDoc : scoreDocs) {
+ int id = getId(searcher, scoreDoc.doc);
+
+ // Check that returned document is not deleted
+ assertFalse(id >= startIndex && id <= endIndex);
+ }
+ // Check that all live docs are returned
+ assertEquals(numDocs - endIndex + startIndex - 1, scoreDocs.length);
+ }
+ }
+ }
+
+ public void testAllDeletes() throws IOException {
+ try (Directory dir = getIndexStore(getRandomVectors(numDocs, dim));
+ IndexWriter w = new IndexWriter(dir, newIndexWriterConfig())) {
+ // Delete all documents
+ w.deleteDocuments(MatchAllDocsQuery.INSTANCE);
+ w.commit();
+
+ try (IndexReader reader = DirectoryReader.open(dir)) {
+ IndexSearcher searcher = newSearcher(reader);
+
+ Query query =
+ getVectorQuery(
+ vectorField,
+ getRandomVector(dim),
+ Float.NEGATIVE_INFINITY,
+ Float.NEGATIVE_INFINITY,
+ null);
+
+ // Check that no vectors are found
+ assertEquals(0, searcher.count(query));
+ }
+ }
+ }
+
+ public void testBoostQuery() throws IOException {
+ // Define the boost and allowed delta
+ float boost = random().nextFloat(5, 10);
+ float delta = 1e-3f;
+
+ try (Directory indexStore = getIndexStore(getRandomVectors(numDocs, dim));
+ IndexReader reader = DirectoryReader.open(indexStore)) {
+ IndexSearcher searcher = newSearcher(reader);
+
+ Query query1 =
+ getVectorQuery(
+ vectorField,
+ getRandomVector(dim),
+ Float.NEGATIVE_INFINITY,
+ Float.NEGATIVE_INFINITY,
+ null);
+ ScoreDoc[] scoreDocs1 = searcher.search(query1, numDocs).scoreDocs;
+
+ Query query2 = new BoostQuery(query1, boost);
+ ScoreDoc[] scoreDocs2 = searcher.search(query2, numDocs).scoreDocs;
+
+ // Check that original scores and boosted scores are equal considering the delta to account
+ // for floating point precision limitations. Don't take the exact result order into
+ // consideration as for small original scores with tiny differences
+ // the boosted scores can be become the same, which might affect the result order.
+ assertEquals(scoreDocs1.length, scoreDocs2.length);
+ for (int i = 0; i < scoreDocs1.length; i++) {
+ int idx = i;
+ Optional<ScoreDoc> boostedDoc =
+ Arrays.stream(scoreDocs2).filter(d -> d.doc == scoreDocs1[idx].doc).findFirst();
+
+ assertTrue(boostedDoc.isPresent());
+ assertEquals(boost * scoreDocs1[i].score, boostedDoc.get().score, delta);
+ }
+ }
+ }
+
+ public void testVectorsAboveSimilarity() throws IOException {
+ // Pick number of docs to accept
+ int numAccepted = random().nextInt(numDocs / 3, numDocs / 2);
+ float delta = 1e-3f;
+
+ V[] vectors = getRandomVectors(numDocs, dim);
+ V queryVector = getRandomVector(dim);
+
+ // Find score above which we get (at least) numAccepted vectors
+ float resultSimilarity = getSimilarity(vectors, queryVector, numAccepted);
+
+ // Cache scores of vectors
+ Map<Integer, Float> scores = new HashMap<>();
+ for (int i = 0; i < numDocs; i++) {
+ float score = compare(queryVector, vectors[i]);
+ if (score >= resultSimilarity) {
+ scores.put(i, score);
+ }
+ }
+
+ try (Directory indexStore = getIndexStore(vectors);
+ IndexReader reader = DirectoryReader.open(indexStore)) {
+ IndexSearcher searcher = newSearcher(reader);
+
+ Query query =
+ getVectorQuery(vectorField, queryVector, Float.NEGATIVE_INFINITY, resultSimilarity, null);
+
+ ScoreDoc[] scoreDocs = searcher.search(query, numDocs).scoreDocs;
+ for (ScoreDoc scoreDoc : scoreDocs) {
+ int id = getId(searcher, scoreDoc.doc);
+
+ // Check that the collected result is above accepted similarity
+ assertTrue(scores.containsKey(id));
+
+ // Check that the score is correct
+ assertEquals(scores.get(id), scoreDoc.score, delta);
+ }
+
+ // Check that all results are collected
+ assertEquals(scores.size(), scoreDocs.length);
+ }
+ }
+
+ public void testFallbackToExact() throws IOException {
+ // Restrictive filter, along with similarity to visit a large number of nodes
+ int numFiltered = numDocs / 5;
+ int targetVisited = numDocs;
+
+ V[] vectors = getRandomVectors(numDocs, dim);
+ V queryVector = getRandomVector(dim);
+
+ float resultSimilarity = getSimilarity(vectors, queryVector, targetVisited);
+ Query filter = IntField.newSetQuery(idField, getFiltered(numFiltered));
+
+ try (Directory indexStore = getIndexStore(vectors);
+ IndexReader reader = DirectoryReader.open(indexStore)) {
+ IndexSearcher searcher = newSearcher(reader);
+
+ Query query =
+ getThrowingVectorQuery(
+ vectorField, queryVector, resultSimilarity, resultSimilarity, filter);
+
+ // Falls back to exact search
+ expectThrows(UnsupportedOperationException.class, () -> searcher.count(query));
+ }
+ }
+
+ public void testApproximate() throws IOException {
+ // Non-restrictive filter, along with similarity to visit a small number of nodes
+ int numFiltered = numDocs - 1;
+ int targetVisited = random().nextInt(1, numFiltered / 10);
+
+ V[] vectors = getRandomVectors(numDocs, dim);
+ V queryVector = getRandomVector(dim);
+
+ float resultSimilarity = getSimilarity(vectors, queryVector, targetVisited);
+ Query filter = IntField.newSetQuery(idField, getFiltered(numFiltered));
+
+ try (Directory indexStore = getIndexStore(vectors);
+ IndexWriter w =
+ new IndexWriter(
+ indexStore,
+ newIndexWriterConfig()
+ .setCodec(
+ TestUtil.alwaysKnnVectorsFormat(
+ new Lucene99HnswVectorsFormat(
+ DEFAULT_MAX_CONN, DEFAULT_BEAM_WIDTH, 0))))) {
+ // Force merge because smaller segments have few filtered docs and often fall back to exact
+ // search, making this test flaky
+ w.forceMerge(1);
+ w.commit();
+
+ try (IndexReader reader = DirectoryReader.open(indexStore)) {
+ IndexSearcher searcher = newSearcher(reader);
+
+ Query query =
+ getThrowingVectorQuery(
+ vectorField, queryVector, resultSimilarity, resultSimilarity, filter);
+
+ // Does not fall back to exact search
+ assertTrue(searcher.count(query) <= numFiltered);
+ }
+ }
+ }
+
+ /** Test that the query times out correctly. */
+ public void testTimeout() throws IOException {
+ V[] vectors = getRandomVectors(numDocs, dim);
+ V queryVector = getRandomVector(dim);
+
+ try (Directory indexStore = getIndexStore(vectors);
+ IndexReader reader = DirectoryReader.open(indexStore)) {
+ IndexSearcher searcher = newSearcher(reader);
+
+ // This query is cacheable, explicitly prevent it
+ searcher.setQueryCache(null);
+
+ Query query =
+ new CountingQuery(
+ getVectorQuery(
+ vectorField,
+ queryVector,
+ Float.NEGATIVE_INFINITY,
+ Float.NEGATIVE_INFINITY,
+ null));
+
+ assertEquals(numDocs, searcher.count(query)); // Expect some results without timeout
+
+ searcher.setTimeout(() -> true); // Immediately timeout
+ assertEquals(0, searcher.count(query)); // Expect no results with the timeout
+
+ searcher.setTimeout(new CountingQueryTimeout(numDocs - 1)); // Do not score all docs
+ int count = searcher.count(query);
+ assertTrue(
+ "0 < count=" + count + " < numDocs=" + numDocs,
+ count > 0 && count < numDocs); // Expect partial results
+
+ // Test timeout with filter
+ int numFiltered = random().nextInt(numDocs / 2, numDocs);
+ Query filter = IntField.newSetQuery(idField, getFiltered(numFiltered));
+ Query filteredQuery =
+ new CountingQuery(
+ getVectorQuery(
+ vectorField,
+ queryVector,
+ Float.NEGATIVE_INFINITY,
+ Float.NEGATIVE_INFINITY,
+ filter));
+
+ searcher.setTimeout(() -> false); // Set a timeout which is never met
+ assertEquals(numFiltered, searcher.count(filteredQuery));
+
+ searcher.setTimeout(
+ new CountingQueryTimeout(numFiltered - 1)); // Timeout before scoring all filtered docs
+ int filteredCount = searcher.count(filteredQuery);
+ assertTrue(
+ "0 < filteredCount=" + filteredCount + " < numFiltered=" + numFiltered,
+ filteredCount > 0 && filteredCount < numFiltered); // Expect partial results
+ }
+ }
+
+ private float getSimilarity(V[] vectors, V queryVector, int targetVisited) {
+ assertTrue(targetVisited >= 0 && targetVisited <= numDocs);
+ if (targetVisited == 0) {
+ return Float.POSITIVE_INFINITY;
+ }
+
+ float[] scores = new float[numDocs];
+ for (int i = 0; i < numDocs; i++) {
+ scores[i] = compare(queryVector, vectors[i]);
+ }
+ Arrays.sort(scores);
+
+ return scores[numDocs - targetVisited];
+ }
+
+ private int[] getFiltered(int numFiltered) {
+ Set<Integer> accepted = new HashSet<>();
+ for (int i = 0; i < numFiltered; ) {
+ int index = random().nextInt(numDocs);
+ if (!accepted.contains(index)) {
+ accepted.add(index);
+ i++;
+ }
+ }
+ return accepted.stream().mapToInt(Integer::intValue).toArray();
+ }
+
+ private int getId(IndexSearcher searcher, int doc) throws IOException {
+ return Objects.requireNonNull(searcher.storedFields().document(doc).getField(idField))
+ .numericValue()
+ .intValue();
+ }
+
+ @SuppressWarnings("unchecked")
+ V[] getRandomVectors(int numDocs, int dim) {
+ return (V[]) IntStream.range(0, numDocs).mapToObj(i -> getRandomVector(dim)).toArray();
+ }
+
+ @SafeVarargs
+ final Directory getIndexStore(V... vectors) throws IOException {
+ Directory dir = newDirectory();
+ try (RandomIndexWriter writer =
+ new RandomIndexWriter(
+ random(),
+ dir,
+ newIndexWriterConfig()
+ .setCodec(
+ TestUtil.alwaysKnnVectorsFormat(
+ new Lucene99HnswVectorsFormat(DEFAULT_MAX_CONN, DEFAULT_BEAM_WIDTH, 0))))) {
+ for (int i = 0; i < vectors.length; ++i) {
+ Document doc = new Document();
+ doc.add(getVectorField(vectorField, vectors[i], function));
+ doc.add(new IntField(idField, i, Field.Store.YES));
+ writer.addDocument(doc);
+ }
+ }
+ return dir;
+ }
+
+ private static class CountingQueryTimeout implements QueryTimeout {
+ private int remaining;
+
+ public CountingQueryTimeout(int count) {
+ remaining = count;
+ }
+
+ @Override
+ public boolean shouldExit() {
+ if (remaining > 0) {
+ remaining--;
+ return false;
+ }
+ return true;
+ }
+ }
+
+ /**
+ * A {@link Query} that emulates {@link Weight#count(LeafReaderContext)} by counting number of
+ * docs of underlying {@link Scorer#iterator()}. TODO: This is a workaround to count partial
+ * results of {@link #delegate} because {@link TimeLimitingBulkScorer} immediately discards
+ * results after timeout.
+ */
+ private static class CountingQuery extends Query {
+ private final Query delegate;
+
+ private CountingQuery(Query delegate) {
+ this.delegate = delegate;
+ }
+
+ @Override
+ public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost)
+ throws IOException {
+ return new Weight(this) {
+ final Weight delegateWeight = delegate.createWeight(searcher, scoreMode, boost);
+
+ @Override
+ public Explanation explain(LeafReaderContext context, int doc) throws IOException {
+ return delegateWeight.explain(context, doc);
+ }
+
+ @Override
+ public ScorerSupplier scorerSupplier(LeafReaderContext context) throws IOException {
+ return delegateWeight.scorerSupplier(context);
+ }
+
+ @Override
+ public int count(LeafReaderContext context) throws IOException {
+ Scorer scorer = scorer(context);
+ if (scorer == null) {
+ return 0;
+ }
+
+ int count = 0;
+ DocIdSetIterator iterator = scorer.iterator();
+ while (iterator.nextDoc() != NO_MORE_DOCS) {
+ count++;
+ }
+ return count;
+ }
+
+ @Override
+ public boolean isCacheable(LeafReaderContext ctx) {
+ return delegateWeight.isCacheable(ctx);
+ }
+ };
+ }
+
+ @Override
+ public String toString(String field) {
+ return String.format(
+ Locale.ROOT, "%s[%s]", getClass().getSimpleName(), delegate.toString(field));
+ }
+
+ @Override
+ public void visit(QueryVisitor visitor) {
+ visitor.visitLeaf(this);
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ return sameClassAs(obj) && delegate.equals(((CountingQuery) obj).delegate);
+ }
+
+ @Override
+ public int hashCode() {
+ return delegate.hashCode();
+ }
+ }
+}
diff --git a/lucene/core/src/test/org/apache/lucene/search/TestByteVectorSimilarityQuery.java b/lucene/core/src/test/org/apache/lucene/search/TestByteVectorSimilarityQuery.java
index 307e86d577f..ba63451a952 100644
--- a/lucene/core/src/test/org/apache/lucene/search/TestByteVectorSimilarityQuery.java
+++ b/lucene/core/src/test/org/apache/lucene/search/TestByteVectorSimilarityQuery.java
@@ -60,7 +60,7 @@ public class TestByteVectorSimilarityQuery
@Override
ByteVectorSimilarityQuery getVectorQuery(
String field, byte[] vector, float resultSimilarity, float decay, Query filter) {
- return new ByteVectorSimilarityQuery(field, vector, resultSimilarity, decay, filter);
+ return new ByteVectorSimilarityQuery.Adaptive(field, vector, resultSimilarity, decay, filter);
}
@Override
@@ -71,14 +71,14 @@ public class TestByteVectorSimilarityQuery
float decay,
Query filter,
KnnSearchStrategy searchStrategy) {
- return new ByteVectorSimilarityQuery(
+ return new ByteVectorSimilarityQuery.Adaptive(
field, vector, resultSimilarity, decay, filter, searchStrategy);
}
@Override
ByteVectorSimilarityQuery getThrowingVectorQuery(
String field, byte[] vector, float resultSimilarity, float decay, Query filter) {
- return new ByteVectorSimilarityQuery(field, vector, resultSimilarity, decay, filter) {
+ return new ByteVectorSimilarityQuery.Adaptive(field, vector, resultSimilarity, decay, filter) {
@Override
VectorScorer createVectorScorer(LeafReaderContext context) {
throw new UnsupportedOperationException();
diff --git a/lucene/core/src/test/org/apache/lucene/search/TestByteVectorSimilarityQuery.java b/lucene/core/src/test/org/apache/lucene/search/TestExplicitByteVectorSimilarityQuery.java
similarity index 75%
copy from lucene/core/src/test/org/apache/lucene/search/TestByteVectorSimilarityQuery.java
copy to lucene/core/src/test/org/apache/lucene/search/TestExplicitByteVectorSimilarityQuery.java
index 307e86d577f..83f9da7f28b 100644
--- a/lucene/core/src/test/org/apache/lucene/search/TestByteVectorSimilarityQuery.java
+++ b/lucene/core/src/test/org/apache/lucene/search/TestExplicitByteVectorSimilarityQuery.java
@@ -20,12 +20,12 @@ import java.util.Arrays;
import org.apache.lucene.document.KnnByteVectorField;
import org.apache.lucene.index.LeafReaderContext;
import org.apache.lucene.index.VectorSimilarityFunction;
-import org.apache.lucene.search.knn.KnnSearchStrategy;
import org.apache.lucene.util.TestVectorUtil;
import org.junit.Before;
-public class TestByteVectorSimilarityQuery
- extends BaseVectorSimilarityQueryTestCase<
+@Deprecated
+public class TestExplicitByteVectorSimilarityQuery
+ extends BaseExplicitVectorSimilarityQueryTestCase<
byte[], KnnByteVectorField, ByteVectorSimilarityQuery> {
@Before
@@ -57,28 +57,26 @@ public class TestByteVectorSimilarityQuery
return new KnnByteVectorField(name, vector, function);
}
- @Override
- ByteVectorSimilarityQuery getVectorQuery(
- String field, byte[] vector, float resultSimilarity, float decay, Query filter) {
- return new ByteVectorSimilarityQuery(field, vector, resultSimilarity, decay, filter);
- }
-
@Override
ByteVectorSimilarityQuery getVectorQuery(
String field,
byte[] vector,
+ float traversalSimilarity,
float resultSimilarity,
- float decay,
- Query filter,
- KnnSearchStrategy searchStrategy) {
- return new ByteVectorSimilarityQuery(
- field, vector, resultSimilarity, decay, filter, searchStrategy);
+ Query filter) {
+ return new ByteVectorSimilarityQuery.Explicit(
+ field, vector, traversalSimilarity, resultSimilarity, filter);
}
@Override
ByteVectorSimilarityQuery getThrowingVectorQuery(
- String field, byte[] vector, float resultSimilarity, float decay, Query filter) {
- return new ByteVectorSimilarityQuery(field, vector, resultSimilarity, decay, filter) {
+ String field,
+ byte[] vector,
+ float traversalSimilarity,
+ float resultSimilarity,
+ Query filter) {
+ return new ByteVectorSimilarityQuery.Explicit(
+ field, vector, traversalSimilarity, resultSimilarity, filter) {
@Override
VectorScorer createVectorScorer(LeafReaderContext context) {
throw new UnsupportedOperationException();
diff --git a/lucene/core/src/test/org/apache/lucene/search/TestFloatVectorSimilarityQuery.java b/lucene/core/src/test/org/apache/lucene/search/TestExplicitFloatVectorSimilarityQuery.java
similarity index 75%
copy from lucene/core/src/test/org/apache/lucene/search/TestFloatVectorSimilarityQuery.java
copy to lucene/core/src/test/org/apache/lucene/search/TestExplicitFloatVectorSimilarityQuery.java
index 2a82fd00c89..bb12318e366 100644
--- a/lucene/core/src/test/org/apache/lucene/search/TestFloatVectorSimilarityQuery.java
+++ b/lucene/core/src/test/org/apache/lucene/search/TestExplicitFloatVectorSimilarityQuery.java
@@ -20,12 +20,12 @@ import java.util.Arrays;
import org.apache.lucene.document.KnnFloatVectorField;
import org.apache.lucene.index.LeafReaderContext;
import org.apache.lucene.index.VectorSimilarityFunction;
-import org.apache.lucene.search.knn.KnnSearchStrategy;
import org.apache.lucene.util.TestVectorUtil;
import org.junit.Before;
-public class TestFloatVectorSimilarityQuery
- extends BaseVectorSimilarityQueryTestCase<
+@Deprecated
+public class TestExplicitFloatVectorSimilarityQuery
+ extends BaseExplicitVectorSimilarityQueryTestCase<
float[], KnnFloatVectorField, FloatVectorSimilarityQuery> {
@Before
@@ -58,28 +58,26 @@ public class TestFloatVectorSimilarityQuery
return new KnnFloatVectorField(name, vector, function);
}
- @Override
- FloatVectorSimilarityQuery getVectorQuery(
- String field, float[] vector, float resultSimilarity, float decay, Query filter) {
- return new FloatVectorSimilarityQuery(field, vector, resultSimilarity, decay, filter);
- }
-
@Override
FloatVectorSimilarityQuery getVectorQuery(
String field,
float[] vector,
+ float traversalSimilarity,
float resultSimilarity,
- float decay,
- Query filter,
- KnnSearchStrategy searchStrategy) {
- return new FloatVectorSimilarityQuery(
- field, vector, resultSimilarity, decay, filter, searchStrategy);
+ Query filter) {
+ return new FloatVectorSimilarityQuery.Explicit(
+ field, vector, traversalSimilarity, resultSimilarity, filter);
}
@Override
FloatVectorSimilarityQuery getThrowingVectorQuery(
- String field, float[] vector, float resultSimilarity, float decay, Query filter) {
- return new FloatVectorSimilarityQuery(field, vector, resultSimilarity, decay, filter) {
+ String field,
+ float[] vector,
+ float traversalSimilarity,
+ float resultSimilarity,
+ Query filter) {
+ return new FloatVectorSimilarityQuery.Explicit(
+ field, vector, traversalSimilarity, resultSimilarity, filter) {
@Override
VectorScorer createVectorScorer(LeafReaderContext context) {
throw new UnsupportedOperationException();
diff --git a/lucene/core/src/test/org/apache/lucene/search/TestFloatVectorSimilarityQuery.java b/lucene/core/src/test/org/apache/lucene/search/TestFloatVectorSimilarityQuery.java
index 2a82fd00c89..93a2b11f0ce 100644
--- a/lucene/core/src/test/org/apache/lucene/search/TestFloatVectorSimilarityQuery.java
+++ b/lucene/core/src/test/org/apache/lucene/search/TestFloatVectorSimilarityQuery.java
@@ -61,7 +61,7 @@ public class TestFloatVectorSimilarityQuery
@Override
FloatVectorSimilarityQuery getVectorQuery(
String field, float[] vector, float resultSimilarity, float decay, Query filter) {
- return new FloatVectorSimilarityQuery(field, vector, resultSimilarity, decay, filter);
+ return new FloatVectorSimilarityQuery.Adaptive(field, vector, resultSimilarity, decay, filter);
}
@Override
@@ -72,14 +72,14 @@ public class TestFloatVectorSimilarityQuery
float decay,
Query filter,
KnnSearchStrategy searchStrategy) {
- return new FloatVectorSimilarityQuery(
+ return new FloatVectorSimilarityQuery.Adaptive(
field, vector, resultSimilarity, decay, filter, searchStrategy);
}
@Override
FloatVectorSimilarityQuery getThrowingVectorQuery(
String field, float[] vector, float resultSimilarity, float decay, Query filter) {
- return new FloatVectorSimilarityQuery(field, vector, resultSimilarity, decay, filter) {
+ return new FloatVectorSimilarityQuery.Adaptive(field, vector, resultSimilarity, decay, filter) {
@Override
VectorScorer createVectorScorer(LeafReaderContext context) {
throw new UnsupportedOperationException();
diff --git a/lucene/core/src/test/org/apache/lucene/search/TestVectorSimilarityCollector.java b/lucene/core/src/test/org/apache/lucene/search/TestVectorSimilarityCollector.java
index 083387f1732..e0561abfd9a 100644
--- a/lucene/core/src/test/org/apache/lucene/search/TestVectorSimilarityCollector.java
+++ b/lucene/core/src/test/org/apache/lucene/search/TestVectorSimilarityCollector.java
@@ -49,6 +49,41 @@ public class TestVectorSimilarityCollector extends LuceneTestCase {
assertArrayEquals(new float[] {0.5f, 0.6f, 0.9f, 0.7f, 0.8f}, resultScores, 1e-3f);
}
+ @Deprecated
+ public void testExplicitResultCollection() {
+ float traversalSimilarity = 0.3f, resultSimilarity = 0.5f;
+
+ ExplicitVectorSimilarityCollector collector =
+ new ExplicitVectorSimilarityCollector(
+ traversalSimilarity, resultSimilarity, Integer.MAX_VALUE);
+ int[] nodes = {1, 5, 10, 4, 8, 3, 2, 6, 7, 9};
+ float[] scores = {0.1f, 0.2f, 0.3f, 0.5f, 0.2f, 0.6f, 0.9f, 0.3f, 0.7f, 0.8f};
+
+ float[] minCompetitiveSimilarities = new float[nodes.length];
+ for (int i = 0; i < nodes.length; i++) {
+ collector.collect(nodes[i], scores[i]);
+ minCompetitiveSimilarities[i] = collector.minCompetitiveSimilarity();
+ }
+
+ ScoreDoc[] scoreDocs = collector.topDocs().scoreDocs;
+ int[] resultNodes = new int[scoreDocs.length];
+ float[] resultScores = new float[scoreDocs.length];
+ for (int i = 0; i < scoreDocs.length; i++) {
+ resultNodes[i] = scoreDocs[i].doc;
+ resultScores[i] = scoreDocs[i].score;
+ }
+
+ // All nodes above resultSimilarity appear in order of collection
+ assertArrayEquals(new int[] {4, 3, 2, 7, 9}, resultNodes);
+ assertArrayEquals(new float[] {0.5f, 0.6f, 0.9f, 0.7f, 0.8f}, resultScores, 1e-3f);
+
+ // Min competitive similarity is minimum of traversalSimilarity or best result encountered
+ assertArrayEquals(
+ new float[] {0.1f, 0.2f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f, 0.3f},
+ minCompetitiveSimilarities,
+ 1e-3f);
+ }
+
public void testDefaultConstructorUsesDefaultStrategy() {
VectorSimilarityCollector collector =
new VectorSimilarityCollector(0.5f, DEFAULT_DECAY, Integer.MAX_VALUE);