You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "kaivalnp (via GitHub)" <gi...@apache.org> on 2026/09/19 01:59:34 UTC
[PR] Use quantized vectors for fp16 HNSW graph merging [lucene]
kaivalnp opened a new pull request, #16687:
URL: https://github.com/apache/lucene/pull/16687
### Description
Scalar quantization support for `FLOAT16` vectors was recently added in #16473.
During merging of `FLOAT32` vectors, the quantized version is used to build HNSW graphs.
Fixing the `FLOAT16` path to be consistent with `FLOAT32`, instead of using the input vectors during merge, which can be memory intensive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Use quantized vectors for fp16 HNSW graph merging [lucene]
Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on code in PR #16687:
URL: https://github.com/apache/lucene/pull/16687#discussion_r4148629408
##########
lucene/core/src/test/org/apache/lucene/codecs/lucene104/TestLucene104HnswScalarQuantizedVectorsFormat.java:
##########
@@ -311,6 +329,164 @@ public void testSimpleOffHeapSize() throws IOException {
}
}
+ /**
+ * Verifies where merging gets the quantized scorer supplier used to build the HNSW graph: from
+ * query data the flat writer prepared ({@link
+ * Lucene104ScalarQuantizedVectorsWriter#mergeOneFlatVectorFieldForMergeScorer}) for asymmetric
+ * encodings, otherwise from {@link
+ * Lucene104ScalarQuantizedVectorsReader#getRandomVectorScorerSupplierForMerge}.
+ */
+ public void testMergeScorer() throws IOException {
+ int dim = 8;
+
+ for (ScalarEncoding scalarEncoding : ScalarEncoding.values()) {
+ for (VectorEncoding vectorEncoding : VectorEncoding.values()) {
+ if (vectorEncoding.isFloatingPoint() == false) { // not applicable for BYTE
+ continue;
+ }
+
+ MergeScorerCounts counts = new MergeScorerCounts();
+ IndexWriterConfig config =
+ newIndexWriterConfig()
+ .setCodec(
+ TestUtil.alwaysKnnVectorsFormat(
+ new MergeScorerCountingFormat(scalarEncoding, counts)))
+ .setMergeScheduler(new SerialMergeScheduler())
+ .setMergePolicy(NoMergePolicy.INSTANCE); // no merges while indexing
+
+ try (Directory dir = newDirectory();
+ IndexWriter w = new IndexWriter(dir, config)) {
+ for (int i = 0; i < 2; i++) {
+ Document document = new Document();
+
+ IndexableField field =
+ switch (vectorEncoding) {
+ case BYTE -> new KnnByteVectorField("v", randomVector8(dim), DOT_PRODUCT);
Review Comment:
Nice catch, updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Use quantized vectors for fp16 HNSW graph merging [lucene]
Posted by "Pulkitg64 (via GitHub)" <gi...@apache.org>.
Pulkitg64 commented on code in PR #16687:
URL: https://github.com/apache/lucene/pull/16687#discussion_r4148453881
##########
lucene/core/src/test/org/apache/lucene/codecs/lucene104/TestLucene104HnswScalarQuantizedVectorsFormat.java:
##########
@@ -311,6 +329,164 @@ public void testSimpleOffHeapSize() throws IOException {
}
}
+ /**
+ * Verifies where merging gets the quantized scorer supplier used to build the HNSW graph: from
+ * query data the flat writer prepared ({@link
+ * Lucene104ScalarQuantizedVectorsWriter#mergeOneFlatVectorFieldForMergeScorer}) for asymmetric
+ * encodings, otherwise from {@link
+ * Lucene104ScalarQuantizedVectorsReader#getRandomVectorScorerSupplierForMerge}.
+ */
+ public void testMergeScorer() throws IOException {
+ int dim = 8;
+
+ for (ScalarEncoding scalarEncoding : ScalarEncoding.values()) {
+ for (VectorEncoding vectorEncoding : VectorEncoding.values()) {
+ if (vectorEncoding.isFloatingPoint() == false) { // not applicable for BYTE
+ continue;
+ }
+
+ MergeScorerCounts counts = new MergeScorerCounts();
+ IndexWriterConfig config =
+ newIndexWriterConfig()
+ .setCodec(
+ TestUtil.alwaysKnnVectorsFormat(
+ new MergeScorerCountingFormat(scalarEncoding, counts)))
+ .setMergeScheduler(new SerialMergeScheduler())
+ .setMergePolicy(NoMergePolicy.INSTANCE); // no merges while indexing
+
+ try (Directory dir = newDirectory();
+ IndexWriter w = new IndexWriter(dir, config)) {
+ for (int i = 0; i < 2; i++) {
+ Document document = new Document();
+
+ IndexableField field =
+ switch (vectorEncoding) {
+ case BYTE -> new KnnByteVectorField("v", randomVector8(dim), DOT_PRODUCT);
Review Comment:
Should we throw error in this branch because we don't expect BYTE vectorEncoding?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Use quantized vectors for fp16 HNSW graph merging [lucene]
Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on PR #16687:
URL: https://github.com/apache/lucene/pull/16687#issuecomment-5879229410
There were some merge conflicts after recent commits, rebased now.
Also added a new test, which fails on `main` with:
```
FLOAT16 x UNSIGNED_BYTE: calls to getRandomVectorScorerSupplierForMerge
Expected :1
Actual :0
```
..and passes with this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Use quantized vectors for fp16 HNSW graph merging [lucene]
Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp merged PR #16687:
URL: https://github.com/apache/lucene/pull/16687
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org