You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Pulkitg64 (via GitHub)" <gi...@apache.org> on 2025/06/16 11:30:44 UTC
[PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Pulkitg64 opened a new pull request, #14792:
URL: https://github.com/apache/lucene/pull/14792
### Description
Introduce `getQuantizedVectorValues` method in `LeafReader` to access `QuantizedVectorValues`.
In a search architecture where searchers and writer runs on separate machine, it is wasteful to have raw float vectors on machine when vector quantization enabled. This PR is adding getQuantizedVectorValues in LeafReader which will help to read QuantizedByteVectors directly without need of reading raw float vectors.
Partially solving #13158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14792:
URL: https://github.com/apache/lucene/pull/14792#issuecomment-2976186238
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "Pulkitg64 (via GitHub)" <gi...@apache.org>.
Pulkitg64 commented on code in PR #14792:
URL: https://github.com/apache/lucene/pull/14792#discussion_r2168151582
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99ScalarQuantizedVectorsReader.java:
##########
@@ -189,6 +190,21 @@ private FieldEntry getFieldEntry(String field) {
public FloatVectorValues getFloatVectorValues(String field) throws IOException {
final FieldEntry fieldEntry = getFieldEntry(field);
final FloatVectorValues rawVectorValues = rawVectorsReader.getFloatVectorValues(field);
+ if (rawVectorValues instanceof OffHeapFloatVectorValues.EmptyOffHeapVectorValues
Review Comment:
Yes, that simplifies things. Fixed in new revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "Pulkitg64 (via GitHub)" <gi...@apache.org>.
Pulkitg64 commented on PR #14792:
URL: https://github.com/apache/lucene/pull/14792#issuecomment-2977859146
We are experimenting with large vector indexes, and since (raw unquantized) vectors consume significant disk space (4x more than quantized vectors), we want to drop the raw vectors from searcher machines. We are currently using vector values for below use cases:
1. Calculating the dot-product scores and return them in search results
2. Returning the vectors in search results
3. Vector counting for metrics
For use case 1 we have started to use [vectorScorer](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99ScalarQuantizedVectorsReader.java#L444) which use quantized vectors for computing score so we are good there. For use cases 2 and 3, we currently use floatVectorValues using [getFloatVectorValues](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99ScalarQuantizedVectorsReader.java#L189) but need to switch to quantizedVectorValues since searchers won't have float vectors anymore and we are okay in accepting the accuracy loss from float-to-byte quantization.
To address these use cases, we have two options:
* Introduce a new API: getQuantizedVectorValues to access quantizedByteVector OR
* Use our local workaround: Make the [QuantizedVectorValues](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99ScalarQuantizedVectorsReader.java#L408) class and its members public to directly access quantized vectors
I would like to know your thoughts on whether we should create such an API, and if you think the above use cases don't justify a new API, what are your thoughts on implementing the workaround solution and pushing it upstream?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "Pulkitg64 (via GitHub)" <gi...@apache.org>.
Pulkitg64 commented on PR #14792:
URL: https://github.com/apache/lucene/pull/14792#issuecomment-2999895760
I have made the new revision to introduce a new class ```OffHeapQuantizedFloatVectorValues``` which will be returned during `getFloatVectorValues()` call and when there are no raw vectors present but quantized vectors still present in the index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduced OffHeapQuantizedFloatVectorValues class to access float vectors when only quantized byte vectors are available in the index [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #14792:
URL: https://github.com/apache/lucene/pull/14792#issuecomment-3755711387
@Pulkitg64 I would say backport & override/mute for the unsupported old format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #14792:
URL: https://github.com/apache/lucene/pull/14792#issuecomment-2977941605
@Pulkitg64
Basically, I don't think callers should "know" directly if they are hitting quantized vectors or raw.
Requiring the user to pick the right thing seems unnecessary when we have the appropriate interfaces already. Its just all about determining how the format itself knows that its missing the `vec` file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduced OffHeapQuantizedFloatVectorValues class to access float vectors when only quantized byte vectors are available in the index [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #14792:
URL: https://github.com/apache/lucene/pull/14792#issuecomment-3761817545
Yeah +1 to backport this original PR (produce de-quantized vectors if full-precision vector isn't present but quantized vector is), and also https://github.com/apache/lucene/pull/15476 (the test infra to confirm future KNN Codecs on 10.x also preserve this capability).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "Pulkitg64 (via GitHub)" <gi...@apache.org>.
Pulkitg64 commented on PR #14792:
URL: https://github.com/apache/lucene/pull/14792#issuecomment-2983706927
Thanks @benwtrent @msokolov
> Again, I think we should do the nice thing, de-quantize the vectors as the user asks for them.
Sorry, I am surely missing something. But if we can't access the quantized vector then how can we do this? We still need byte vector to rehydrate float vectors, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on code in PR #14792:
URL: https://github.com/apache/lucene/pull/14792#discussion_r2167410847
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene95/OffHeapFloatVectorValues.java:
##########
@@ -270,7 +270,8 @@ public DocIdSetIterator iterator() {
}
}
- private static class EmptyOffHeapVectorValues extends OffHeapFloatVectorValues {
+ /** Empty vector values when there are no vectors stored. */
Review Comment:
"values when" should read "values to be returned when"
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99ScalarQuantizedVectorsReader.java:
##########
@@ -189,6 +190,21 @@ private FieldEntry getFieldEntry(String field) {
public FloatVectorValues getFloatVectorValues(String field) throws IOException {
final FieldEntry fieldEntry = getFieldEntry(field);
final FloatVectorValues rawVectorValues = rawVectorsReader.getFloatVectorValues(field);
+ if (rawVectorValues instanceof OffHeapFloatVectorValues.EmptyOffHeapVectorValues
Review Comment:
I think I'd like this better if we checked `rawVectorValues.size() == 0` instead of having an `instanceof` check. We might change the class, or replace it with an anonymous class or something, but we will always have size=0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #14792:
URL: https://github.com/apache/lucene/pull/14792#issuecomment-2976430071
@Pulkitg64 I don't understand how this is part of https://github.com/apache/lucene/issues/13158
I would have thought the APIs stay the same. Quantization should be able to "rehydrate" the quantized vectors into floating point (or whatever the original values).
So, the segment, depending on what data it has access to, will:
- Return the original doc value floating point vectors
- Rehydrate the quantized values.
Either way, users should still be able to call `float[] vectorValue(int ord)`.
I would think there is a sub-class called `QuantizedFloatVectorValues`, that satisfies the `FloatVectorValues` interface.
But maybe we add an `isApproximate()` or a `extractQuantizedValues()` that returns null, or the `QuantizedFloatVectorValues` interface.
But it is likely useless for the user to have access to the quantized bytes directly as they don't provide much value without knowing how to use them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "Pulkitg64 (via GitHub)" <gi...@apache.org>.
Pulkitg64 commented on PR #14792:
URL: https://github.com/apache/lucene/pull/14792#issuecomment-2977454253
Thanks @benwtrent for the quick review and comments.
Regarding your comment about how this relates to issue #13158 - I agree in a way this PR doesn't directly help create a "read-only" index as mentioned in the issue. Let me clarify the motivation:
This PR addresses a scenario where:
* Raw (unquantized) vectors are removed from the index since they aren't needed for searching
* The architecture has searcher and writer running on separate machines
Currently, there's no way to directly access quantized vectors - we can only access raw vectors. But if raw vectors are dropped, this causes errors. This PR adds methods to access ByteQuantizedVectors in such cases.
As for the usefulness of accessing quantized bytes directly - we have specific use cases, such as returning the vectors themselves when requested in a query.
Please let me know your thoughts.
Regarding accessing quantized vectors directly - we could also consider using the [QuantizedVectorValues](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99ScalarQuantizedVectorsReader.java#L408) class, which is currently returned by the [getFloatVectorValues](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99ScalarQuantizedVectorsReader.java#L189) method. While this class wraps both raw and quantized vectors, its members are private, preventing direct access to the quantized vectors like we're doing in this PR.
Would it make more sense to make the relevant members public in QuantizedVectorValues rather than adding getQuantizedVectorValues to LeafReader?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on code in PR #14792:
URL: https://github.com/apache/lucene/pull/14792#discussion_r2167546639
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene95/OffHeapFloatVectorValues.java:
##########
@@ -270,7 +270,8 @@ public DocIdSetIterator iterator() {
}
}
- private static class EmptyOffHeapVectorValues extends OffHeapFloatVectorValues {
Review Comment:
if switching to `size() == 0` for float vector existence, I think this change is unnecessary.
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene99/OffHeapQuantizedFloatVectorValues.java:
##########
@@ -0,0 +1,365 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.codecs.lucene99;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.lucene.codecs.hnsw.FlatVectorsScorer;
+import org.apache.lucene.codecs.lucene90.IndexedDISI;
+import org.apache.lucene.codecs.lucene95.HasIndexSlice;
+import org.apache.lucene.codecs.lucene95.OrdToDocDISIReaderConfiguration;
+import org.apache.lucene.index.FloatVectorValues;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.search.VectorScorer;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.util.Bits;
+import org.apache.lucene.util.hnsw.RandomVectorScorer;
+import org.apache.lucene.util.packed.DirectMonotonicReader;
+import org.apache.lucene.util.quantization.ScalarQuantizer;
+
+/** Read the vector values from the index input. This supports both iterated and random access. */
+public abstract class OffHeapQuantizedFloatVectorValues extends FloatVectorValues
Review Comment:
If this is gonna be public, it should likely have a `internal` annotation to make folks aware that its interface can break whenever we want.
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene99/OffHeapQuantizedFloatVectorValues.java:
##########
@@ -0,0 +1,365 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.codecs.lucene99;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.lucene.codecs.hnsw.FlatVectorsScorer;
+import org.apache.lucene.codecs.lucene90.IndexedDISI;
+import org.apache.lucene.codecs.lucene95.HasIndexSlice;
+import org.apache.lucene.codecs.lucene95.OrdToDocDISIReaderConfiguration;
+import org.apache.lucene.index.FloatVectorValues;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.search.VectorScorer;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.util.Bits;
+import org.apache.lucene.util.hnsw.RandomVectorScorer;
+import org.apache.lucene.util.packed.DirectMonotonicReader;
+import org.apache.lucene.util.quantization.ScalarQuantizer;
+
+/** Read the vector values from the index input. This supports both iterated and random access. */
+public abstract class OffHeapQuantizedFloatVectorValues extends FloatVectorValues
+ implements HasIndexSlice {
+
+ protected final int dimension;
+ protected final int size;
+ protected final int numBytes;
+ protected final ScalarQuantizer scalarQuantizer;
+ protected final VectorSimilarityFunction similarityFunction;
+ protected final FlatVectorsScorer vectorsScorer;
+ protected final boolean compress;
+
+ protected final IndexInput slice;
+ protected final float[] floatValue;
+ protected final byte[] binaryValue;
+ protected final ByteBuffer byteBuffer;
+ protected final int byteSize;
+ protected int curOrd = -1;
+ protected final float[] scoreCorrectionConstant = new float[1];
+
+ OffHeapQuantizedFloatVectorValues(
+ int dimension,
+ int size,
+ ScalarQuantizer scalarQuantizer,
+ VectorSimilarityFunction similarityFunction,
+ FlatVectorsScorer vectorsScorer,
+ boolean compress,
+ IndexInput slice) {
+ this.dimension = dimension;
+ this.size = size;
+ this.slice = slice;
+ this.scalarQuantizer = scalarQuantizer;
+ this.compress = compress;
+ if (scalarQuantizer.getBits() <= 4 && compress) {
+ this.numBytes = (dimension + 1) >> 1;
+ } else {
+ this.numBytes = dimension;
+ }
+ this.byteSize = this.numBytes + Float.BYTES;
+ byteBuffer = ByteBuffer.allocate(dimension);
+ binaryValue = byteBuffer.array();
+ floatValue = new float[dimension];
+ this.similarityFunction = similarityFunction;
+ this.vectorsScorer = vectorsScorer;
+ }
+
+ public ScalarQuantizer getScalarQuantizer() {
+ return scalarQuantizer;
+ }
Review Comment:
this isn't strictly necessary
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99ScalarQuantizedVectorsReader.java:
##########
@@ -189,6 +190,21 @@ private FieldEntry getFieldEntry(String field) {
public FloatVectorValues getFloatVectorValues(String field) throws IOException {
final FieldEntry fieldEntry = getFieldEntry(field);
final FloatVectorValues rawVectorValues = rawVectorsReader.getFloatVectorValues(field);
+ if (rawVectorValues instanceof OffHeapFloatVectorValues.EmptyOffHeapVectorValues
Review Comment:
I agree with @msokolov here. Checking size simplifies things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14792:
URL: https://github.com/apache/lucene/pull/14792#issuecomment-2998156913
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #14792:
URL: https://github.com/apache/lucene/pull/14792#issuecomment-2983852051
@Pulkitg64 I think the format has access to the quantized vectors. I am saying we shouldn't add a new `getQuantizedVEctors` to the leaf or kNN APIs.
When the `vec` file isn't present, the format should still return a `FloatVectorValues`, and when folks call `float[] getVector(int ord)`, they get the de-quantized representation of the vector.
This should require no new public facing APIs anywhere.
Here is the idea.
```
public FloatVectorValues getFloatVectorValues(String field) throws IOException {
return OffHeapQuantizedFloatVectorValues.load(...);
}
...
static class OffHeapQuantizedFloatVectorValues extends FloatVectorValues {
final float[] scratch;
int curOrd = -1;
...
public float[] vectorValue(int ord) throws IOException {
if (ord == curOrd) return scratch;
dequantize(ord);
curOrd = ord;
return scratch;
}
private void dequantize(int ord) throws IOException {
//read quantized values and parameters
// dequantize into scratch
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "Pulkitg64 (via GitHub)" <gi...@apache.org>.
Pulkitg64 commented on code in PR #14792:
URL: https://github.com/apache/lucene/pull/14792#discussion_r2168100586
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene95/OffHeapFloatVectorValues.java:
##########
@@ -270,7 +270,8 @@ public DocIdSetIterator iterator() {
}
}
- private static class EmptyOffHeapVectorValues extends OffHeapFloatVectorValues {
+ /** Empty vector values when there are no vectors stored. */
Review Comment:
Removed this change, since we will check size==0 instead of making the class public.
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene95/OffHeapFloatVectorValues.java:
##########
@@ -270,7 +270,8 @@ public DocIdSetIterator iterator() {
}
}
- private static class EmptyOffHeapVectorValues extends OffHeapFloatVectorValues {
Review Comment:
Removed this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #14792:
URL: https://github.com/apache/lucene/pull/14792#issuecomment-2977492907
> As for the usefulness of accessing quantized bytes directly - we have specific use cases, such as returning the vectors themselves when requested in a query.
I would assume the caller would want something akin to the `float` values. What would a caller be expected to do with the quantized bytes directly?
I am saying that returning the quantized bytes, without knowing all the other information (quantized technique, the technique's parameters, etc.) is pretty useless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "Pulkitg64 (via GitHub)" <gi...@apache.org>.
Pulkitg64 commented on PR #14792:
URL: https://github.com/apache/lucene/pull/14792#issuecomment-2984235394
Oh I understand now what you meant in your comments. This approach is much more cleaner and doesn't require any new API addition. Will raise a new revision in some time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduced OffHeapQuantizedFloatVectorValues class to access float vectors when only quantized byte vectors are available in the index [lucene]
Posted by "Pulkitg64 (via GitHub)" <gi...@apache.org>.
Pulkitg64 commented on PR #14792:
URL: https://github.com/apache/lucene/pull/14792#issuecomment-3755699438
Hey @benwtrent @msokolov, is it good idea to back port this change to 10.x branch regarding [this](https://github.com/apache/lucene/pull/14792#issuecomment-3745277101) comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #14792:
URL: https://github.com/apache/lucene/pull/14792#issuecomment-2977917618
> Returning the vectors in search results
Why would you need to do this?
Generally, I would assume that any access to the vector would be "Give me what I gave you", and the best we can do with quantized vectors is the dequantized vector.
I don't fully understand how serializing a read-only segment that is missing files (e.g. missing the "vec" file), but the format should do the right thing and see that the file isn't there and provide an approximate view of the floating point vectors.
> Vector counting for metrics
I don't understand what this means really. Just counting how many vectors there are? This should be doable via the `FloatVectorValues` interface.
> but need to switch to quantizedVectorValues since searchers won't have float vectors anymore and we are okay in accepting the accuracy loss from float-to-byte quantization.
Again, I think we should do the nice thing, de-quantize the vectors as the user asks for them.
It should fully satisfy the `FloatVectorValues` API, de-quantizing the vectors and indicate that the vector returned is an approximation.
Getting access to the raw quantized bytes is basically useless without all the other parameters that were used to quantized the vector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on PR #14792:
URL: https://github.com/apache/lucene/pull/14792#issuecomment-2981195460
For the return values use case, another choice is to disable it in the case the original vectors were not "stored" in the searchable index. Otherwise, I agree with Ben that we could support "rehydration" in the codec. For example, suppose we see that we have zero full-precision vectors, but nonzero quantized vectors; then we could fall back to "rehydration".
For the counting case (get total number of vectors), should we always use the quantized count where today we use the full-precision count?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduce getQuantizedVectorValues method in LeafReader to access QuantizedByteVectorValues [lucene]
Posted by "Pulkitg64 (via GitHub)" <gi...@apache.org>.
Pulkitg64 commented on code in PR #14792:
URL: https://github.com/apache/lucene/pull/14792#discussion_r2168100860
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene99/OffHeapQuantizedFloatVectorValues.java:
##########
@@ -0,0 +1,365 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.codecs.lucene99;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import org.apache.lucene.codecs.hnsw.FlatVectorsScorer;
+import org.apache.lucene.codecs.lucene90.IndexedDISI;
+import org.apache.lucene.codecs.lucene95.HasIndexSlice;
+import org.apache.lucene.codecs.lucene95.OrdToDocDISIReaderConfiguration;
+import org.apache.lucene.index.FloatVectorValues;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.search.VectorScorer;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.util.Bits;
+import org.apache.lucene.util.hnsw.RandomVectorScorer;
+import org.apache.lucene.util.packed.DirectMonotonicReader;
+import org.apache.lucene.util.quantization.ScalarQuantizer;
+
+/** Read the vector values from the index input. This supports both iterated and random access. */
+public abstract class OffHeapQuantizedFloatVectorValues extends FloatVectorValues
+ implements HasIndexSlice {
+
+ protected final int dimension;
+ protected final int size;
+ protected final int numBytes;
+ protected final ScalarQuantizer scalarQuantizer;
+ protected final VectorSimilarityFunction similarityFunction;
+ protected final FlatVectorsScorer vectorsScorer;
+ protected final boolean compress;
+
+ protected final IndexInput slice;
+ protected final float[] floatValue;
+ protected final byte[] binaryValue;
+ protected final ByteBuffer byteBuffer;
+ protected final int byteSize;
+ protected int curOrd = -1;
+ protected final float[] scoreCorrectionConstant = new float[1];
+
+ OffHeapQuantizedFloatVectorValues(
+ int dimension,
+ int size,
+ ScalarQuantizer scalarQuantizer,
+ VectorSimilarityFunction similarityFunction,
+ FlatVectorsScorer vectorsScorer,
+ boolean compress,
+ IndexInput slice) {
+ this.dimension = dimension;
+ this.size = size;
+ this.slice = slice;
+ this.scalarQuantizer = scalarQuantizer;
+ this.compress = compress;
+ if (scalarQuantizer.getBits() <= 4 && compress) {
+ this.numBytes = (dimension + 1) >> 1;
+ } else {
+ this.numBytes = dimension;
+ }
+ this.byteSize = this.numBytes + Float.BYTES;
+ byteBuffer = ByteBuffer.allocate(dimension);
+ binaryValue = byteBuffer.array();
+ floatValue = new float[dimension];
+ this.similarityFunction = similarityFunction;
+ this.vectorsScorer = vectorsScorer;
+ }
+
+ public ScalarQuantizer getScalarQuantizer() {
+ return scalarQuantizer;
+ }
Review Comment:
Removed from new revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Introduced OffHeapQuantizedFloatVectorValues class to access float vectors when only quantized byte vectors are available in the index [lucene]
Posted by "Pulkitg64 (via GitHub)" <gi...@apache.org>.
Pulkitg64 commented on PR #14792:
URL: https://github.com/apache/lucene/pull/14792#issuecomment-3745277101
Should we backport this PR to 10.x branch? I am trying to backport this change about [enforcing fallback support for float vector retrieval in quantized KNN vector formats](https://github.com/apache/lucene/pull/15476) but the test case is failing there because the fallback support is not present in the Lucene99ScalarQuantizedVectorsReader.java on 10x branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org