You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "kaivalnp (via GitHub)" <gi...@apache.org> on 2026/08/12 01:33:51 UTC

[PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

kaivalnp opened a new pull request, #16506:
URL: https://github.com/apache/lucene/pull/16506

   ### Description
   
   Add a scalar-quantized version of the de-duplicating vector format `DedupHnswScalarQuantizedVectorsFormat` (follow-up to #15979), which stores each distinct vector once in both raw and scalar quantized form: searches score against the quantized vectors, while raw vectors remain available for exact reads and re-scoring (same as Lucene 10.4 OSQ).
   
   De-duplication works like the raw format: vectors of the same dimension and encoding form a group, and each field maps its document ordinals onto shared group ordinals through a `fieldOrdToGroupOrd` translation.
   
   Files:
   - `.vdd`: Raw vectors and per-field data (same as the flat de-duplicating format).
   - `.vdqm` (metadata): groups with their per-flavor block locations (each block records its own `ScalarEncoding`), followed by per-field entries. See below for "flavor".
   - `.vdqd` (quantized records): one block per `FLOAT32` group and quantization flavor in use, one record per distinct vector (quantized bytes + corrective terms in-line with the Lucene 10.4 OSQ format).
   
   Design choices:
   1. Data-blind quantization: Vectors are quantized against a zero centroid instead of a per-field data-derived centroid.
       1. A quantized record is then a pure function of the raw vector and its flavor (independent of the other vectors in the segment), allowing for sane de-duplication between fields containing overlapping vectors.
       2. This also gives merges a fast path: records from source segments in this format are copied verbatim instead of re-reading raw vectors and re-running the interval optimization (re-quantization only happens on flush, or when a source has an incompatible encoding/flavor).
       3. The tradeoff is accuracy on off-center data relative to centroid-centered quantization.
   2. Per-similarity "flavors": One record can't serve all similarity functions (`EUCLIDEAN` needs the squared norm as its corrective term; `COSINE` needs write-time normalization).
       1. Each group therefore stores one block per flavor in use — `EUCLIDEAN`, `DOT_PRODUCT` + `MAXIMUM_INNER_PRODUCT`, and `COSINE`, so fields whose similarity functions map to the same flavor share records, and every flavor scores through the stock `Lucene104ScalarQuantizedVectorScorer` with no custom score math.
       2. The scorer is a small subclass of the raw format's scorer: doc operations on the field's values, vector operations delegated to the group view, ordinals translated once per comparison (same as the flat format).
   3. Quantized records are 1:1 with group ordinals: Since quantization is lossy, distinct raw vectors can collapse to identical quantized bytes, so a second level of de-duplication is possible in principle.
       1. This was deliberately left out: it would require maintaining a second ordinal mapping (`groupOrd → quantizedOrd`) off-heap, de-duplicating quantized bytes (hash + equality checks on collision), splitting corrective terms into a separate dense block, and writing temp files + read-back during merge for proper de-duplication.
       2. Keeping records 1:1 with group ordinals shares the existing `fieldOrdToGroupOrd` map between raw and quantized data and keeps merges streaming, at the cost of occasionally storing byte-identical quantized records for distinct raw vectors.
   
   **DISCLAIMER**: Much of this code was written by an AI, but I have reviewed and refined most of it!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on code in PR #16506:
URL: https://github.com/apache/lucene/pull/16506#discussion_r3846213224


##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupQuantizer.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import java.io.IOException;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.lucene.codecs.lucene104.OffHeapScalarQuantizedVectorValues;
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.VectorEncoding;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.VectorUtil;
+import org.apache.lucene.util.quantization.OptimizedScalarQuantizer;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * Write-side helper performing <i>data-blind</i> scalar quantization of de-duplicated vectors.
+ *
+ * <p>Quantization assumes input vectors are evenly distributed around the origin, i.e. the centroid
+ * is a zero vector. The quantized record of a vector (packed bytes, optimized intervals, an
+ * additional correction and the component sum) is then a pure function of the raw vector and the
+ * {@link Flavor} derived from the field's similarity function: independent of the other vectors in
+ * the segment. Identical vectors thus quantize identically and the quantized data de-duplicates
+ * along with the raw data, shared across all documents and fields of a group with the same flavor.
+ *
+ * <p>For the same reason, a record already computed by a source segment does not change on merge:
+ * when a distinct vector originates from a segment in this format (with the same encoding and
+ * flavor), its quantized record is <b>copied</b> instead of re-reading the raw vector and
+ * re-quantizing it.
+ *
+ * <p>Records follow the {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsFormat} conventions exactly
+ * (per flavor), so they are scored by the stock {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer}.
+ *
+ * @lucene.experimental
+ */
+record DedupQuantizer(ScalarEncoding encoding) {
+
+  /**
+   * How a vector is prepared and quantized, derived from the field's {@link
+   * VectorSimilarityFunction}. Fields of a group whose similarity functions map to the same flavor
+   * share one quantized record per distinct vector; a group stores one block per flavor in use.
+   */
+  enum Flavor {
+
+    /** Vectors quantized as-is; the additional correction holds the squared norm. */
+    EUCLIDEAN(VectorSimilarityFunction.EUCLIDEAN, false),
+
+    /**
+     * Vectors quantized as-is; the additional correction holds the dot product with the (zero)
+     * centroid, i.e. zero.
+     */
+    DOT_PRODUCT(VectorSimilarityFunction.DOT_PRODUCT, false),
+
+    /**
+     * Vectors are l2-normalized before quantization (for cosine similarity, scored as a dot
+     * product); the additional correction is zero as for {@link #DOT_PRODUCT}.
+     */
+    NORMALIZED(VectorSimilarityFunction.DOT_PRODUCT, true);
+
+    private final VectorSimilarityFunction quantizerFunction;
+    private final boolean normalized;
+
+    Flavor(VectorSimilarityFunction quantizerFunction, boolean normalized) {
+      this.quantizerFunction = quantizerFunction;
+      this.normalized = normalized;
+    }
+
+    static Flavor of(VectorSimilarityFunction function) {
+      return switch (function) {
+        case EUCLIDEAN -> EUCLIDEAN;
+        case DOT_PRODUCT, MAXIMUM_INNER_PRODUCT -> DOT_PRODUCT;
+        case COSINE -> NORMALIZED;
+      };
+    }
+
+    /**
+     * The quantizer producing this flavor's records against a zero centroid. NOTE: {@link
+     * #NORMALIZED} quantizes with {@link VectorSimilarityFunction#DOT_PRODUCT} (identical bytes and
+     * corrections, without the unit-centroid requirement of a cosine-configured quantizer).
+     */
+    OptimizedScalarQuantizer quantizer() {
+      return new OptimizedScalarQuantizer(quantizerFunction);
+    }
+
+    boolean normalized() {
+      return normalized;
+    }
+  }
+
+  /** Supplies the distinct (raw) float vector at a group ordinal. */
+  interface FloatVectorSupplier {

Review Comment:
   This is `package-private`. When we enable quantization on `float16[]`, it will likely need new methods _inside_ this class and the de-duplicating vector writer -- but will _not_ be breaking for end-users who can only consume the top-level format (i.e. `DedupHnswScalarQuantizedVectorsFormat`), which will be unchanged.



##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupQuantizer.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import java.io.IOException;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.lucene.codecs.lucene104.OffHeapScalarQuantizedVectorValues;
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.VectorEncoding;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.VectorUtil;
+import org.apache.lucene.util.quantization.OptimizedScalarQuantizer;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * Write-side helper performing <i>data-blind</i> scalar quantization of de-duplicated vectors.
+ *
+ * <p>Quantization assumes input vectors are evenly distributed around the origin, i.e. the centroid
+ * is a zero vector. The quantized record of a vector (packed bytes, optimized intervals, an
+ * additional correction and the component sum) is then a pure function of the raw vector and the
+ * {@link Flavor} derived from the field's similarity function: independent of the other vectors in
+ * the segment. Identical vectors thus quantize identically and the quantized data de-duplicates
+ * along with the raw data, shared across all documents and fields of a group with the same flavor.
+ *
+ * <p>For the same reason, a record already computed by a source segment does not change on merge:
+ * when a distinct vector originates from a segment in this format (with the same encoding and
+ * flavor), its quantized record is <b>copied</b> instead of re-reading the raw vector and
+ * re-quantizing it.
+ *
+ * <p>Records follow the {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsFormat} conventions exactly
+ * (per flavor), so they are scored by the stock {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer}.
+ *
+ * @lucene.experimental
+ */
+record DedupQuantizer(ScalarEncoding encoding) {
+
+  /**
+   * How a vector is prepared and quantized, derived from the field's {@link
+   * VectorSimilarityFunction}. Fields of a group whose similarity functions map to the same flavor
+   * share one quantized record per distinct vector; a group stores one block per flavor in use.
+   */
+  enum Flavor {
+
+    /** Vectors quantized as-is; the additional correction holds the squared norm. */
+    EUCLIDEAN(VectorSimilarityFunction.EUCLIDEAN, false),
+
+    /**
+     * Vectors quantized as-is; the additional correction holds the dot product with the (zero)
+     * centroid, i.e. zero.
+     */
+    DOT_PRODUCT(VectorSimilarityFunction.DOT_PRODUCT, false),
+
+    /**
+     * Vectors are l2-normalized before quantization (for cosine similarity, scored as a dot
+     * product); the additional correction is zero as for {@link #DOT_PRODUCT}.
+     */
+    NORMALIZED(VectorSimilarityFunction.DOT_PRODUCT, true);
+
+    private final VectorSimilarityFunction quantizerFunction;
+    private final boolean normalized;
+
+    Flavor(VectorSimilarityFunction quantizerFunction, boolean normalized) {
+      this.quantizerFunction = quantizerFunction;
+      this.normalized = normalized;
+    }
+
+    static Flavor of(VectorSimilarityFunction function) {
+      return switch (function) {
+        case EUCLIDEAN -> EUCLIDEAN;
+        case DOT_PRODUCT, MAXIMUM_INNER_PRODUCT -> DOT_PRODUCT;
+        case COSINE -> NORMALIZED;
+      };
+    }
+
+    /**
+     * The quantizer producing this flavor's records against a zero centroid. NOTE: {@link
+     * #NORMALIZED} quantizes with {@link VectorSimilarityFunction#DOT_PRODUCT} (identical bytes and
+     * corrections, without the unit-centroid requirement of a cosine-configured quantizer).
+     */
+    OptimizedScalarQuantizer quantizer() {
+      return new OptimizedScalarQuantizer(quantizerFunction);
+    }
+
+    boolean normalized() {
+      return normalized;
+    }
+  }
+
+  /** Supplies the distinct (raw) float vector at a group ordinal. */
+  interface FloatVectorSupplier {
+    float[] get(int ord) throws IOException;
+  }
+
+  /**
+   * The already-quantized record of a distinct vector, held by a source segment at {@code ord} with
+   * the given flavor.
+   */
+  record PreQuantized(QuantizedByteVectorValues values, Flavor flavor, int ord) {}
+
+  /**
+   * Supplies the {@link PreQuantized} record of the distinct vector at a group ordinal, or {@code
+   * null} when its source segment does not hold one.
+   */
+  interface PreQuantizedSupplier {
+    PreQuantized get(int ord) throws IOException;
+  }
+
+  /** The encoding, offset and size of one flavor's quantized data block. */
+  record QuantizedBlock(
+      ScalarEncoding encoding, long quantizedDataOffset, long quantizedDataSize) {}
+
+  /** Writes an empty flavor-block list, for groups without quantized data. */
+  static void writeEmptyGroup(IndexOutput meta) throws IOException {
+    meta.writeVInt(0);
+  }
+
+  /** Reads the flavor-block list of a group, the counterpart of {@link #writeGroup}. */
+  static Map<Flavor, QuantizedBlock> readGroup(IndexInput meta) throws IOException {
+    int numFlavors = meta.readVInt();
+    Map<Flavor, QuantizedBlock> blocks = new EnumMap<>(Flavor.class);
+    for (int i = 0; i < numFlavors; i++) {
+      int flavorOrd = meta.readVInt();
+      if (flavorOrd < 0 || flavorOrd >= Flavor.values().length) {
+        throw new CorruptIndexException("Invalid flavor ordinal: " + flavorOrd, meta);
+      }
+      Flavor flavor = Flavor.values()[flavorOrd];

Review Comment:
   Makes sense, switched to explicit ordinals!



##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupQuantizer.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import java.io.IOException;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.lucene.codecs.lucene104.OffHeapScalarQuantizedVectorValues;
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.VectorEncoding;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.VectorUtil;
+import org.apache.lucene.util.quantization.OptimizedScalarQuantizer;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * Write-side helper performing <i>data-blind</i> scalar quantization of de-duplicated vectors.
+ *
+ * <p>Quantization assumes input vectors are evenly distributed around the origin, i.e. the centroid
+ * is a zero vector. The quantized record of a vector (packed bytes, optimized intervals, an
+ * additional correction and the component sum) is then a pure function of the raw vector and the
+ * {@link Flavor} derived from the field's similarity function: independent of the other vectors in
+ * the segment. Identical vectors thus quantize identically and the quantized data de-duplicates
+ * along with the raw data, shared across all documents and fields of a group with the same flavor.
+ *
+ * <p>For the same reason, a record already computed by a source segment does not change on merge:
+ * when a distinct vector originates from a segment in this format (with the same encoding and
+ * flavor), its quantized record is <b>copied</b> instead of re-reading the raw vector and
+ * re-quantizing it.
+ *
+ * <p>Records follow the {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsFormat} conventions exactly
+ * (per flavor), so they are scored by the stock {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer}.
+ *
+ * @lucene.experimental
+ */
+record DedupQuantizer(ScalarEncoding encoding) {
+
+  /**
+   * How a vector is prepared and quantized, derived from the field's {@link
+   * VectorSimilarityFunction}. Fields of a group whose similarity functions map to the same flavor
+   * share one quantized record per distinct vector; a group stores one block per flavor in use.
+   */
+  enum Flavor {
+
+    /** Vectors quantized as-is; the additional correction holds the squared norm. */
+    EUCLIDEAN(VectorSimilarityFunction.EUCLIDEAN, false),
+
+    /**
+     * Vectors quantized as-is; the additional correction holds the dot product with the (zero)
+     * centroid, i.e. zero.
+     */
+    DOT_PRODUCT(VectorSimilarityFunction.DOT_PRODUCT, false),
+
+    /**
+     * Vectors are l2-normalized before quantization (for cosine similarity, scored as a dot
+     * product); the additional correction is zero as for {@link #DOT_PRODUCT}.
+     */
+    NORMALIZED(VectorSimilarityFunction.DOT_PRODUCT, true);
+
+    private final VectorSimilarityFunction quantizerFunction;
+    private final boolean normalized;
+
+    Flavor(VectorSimilarityFunction quantizerFunction, boolean normalized) {
+      this.quantizerFunction = quantizerFunction;
+      this.normalized = normalized;
+    }
+
+    static Flavor of(VectorSimilarityFunction function) {
+      return switch (function) {
+        case EUCLIDEAN -> EUCLIDEAN;
+        case DOT_PRODUCT, MAXIMUM_INNER_PRODUCT -> DOT_PRODUCT;
+        case COSINE -> NORMALIZED;
+      };
+    }
+
+    /**
+     * The quantizer producing this flavor's records against a zero centroid. NOTE: {@link
+     * #NORMALIZED} quantizes with {@link VectorSimilarityFunction#DOT_PRODUCT} (identical bytes and
+     * corrections, without the unit-centroid requirement of a cosine-configured quantizer).
+     */
+    OptimizedScalarQuantizer quantizer() {
+      return new OptimizedScalarQuantizer(quantizerFunction);
+    }
+
+    boolean normalized() {
+      return normalized;
+    }
+  }
+
+  /** Supplies the distinct (raw) float vector at a group ordinal. */
+  interface FloatVectorSupplier {
+    float[] get(int ord) throws IOException;
+  }
+
+  /**
+   * The already-quantized record of a distinct vector, held by a source segment at {@code ord} with
+   * the given flavor.
+   */
+  record PreQuantized(QuantizedByteVectorValues values, Flavor flavor, int ord) {}
+
+  /**
+   * Supplies the {@link PreQuantized} record of the distinct vector at a group ordinal, or {@code
+   * null} when its source segment does not hold one.
+   */
+  interface PreQuantizedSupplier {
+    PreQuantized get(int ord) throws IOException;
+  }
+
+  /** The encoding, offset and size of one flavor's quantized data block. */
+  record QuantizedBlock(
+      ScalarEncoding encoding, long quantizedDataOffset, long quantizedDataSize) {}
+
+  /** Writes an empty flavor-block list, for groups without quantized data. */
+  static void writeEmptyGroup(IndexOutput meta) throws IOException {
+    meta.writeVInt(0);
+  }
+
+  /** Reads the flavor-block list of a group, the counterpart of {@link #writeGroup}. */
+  static Map<Flavor, QuantizedBlock> readGroup(IndexInput meta) throws IOException {
+    int numFlavors = meta.readVInt();
+    Map<Flavor, QuantizedBlock> blocks = new EnumMap<>(Flavor.class);
+    for (int i = 0; i < numFlavors; i++) {
+      int flavorOrd = meta.readVInt();
+      if (flavorOrd < 0 || flavorOrd >= Flavor.values().length) {
+        throw new CorruptIndexException("Invalid flavor ordinal: " + flavorOrd, meta);
+      }
+      Flavor flavor = Flavor.values()[flavorOrd];
+      int wireNumber = meta.readVInt();
+      ScalarEncoding encoding =
+          ScalarEncoding.fromWireNumber(wireNumber)
+              .orElseThrow(
+                  () ->
+                      new CorruptIndexException(
+                          "Invalid scalar encoding wire number: " + wireNumber, meta));
+      long offset = meta.readLong();
+      long size = meta.readLong();
+      if (blocks.put(flavor, new QuantizedBlock(encoding, offset, size)) != null) {
+        throw new CorruptIndexException("Duplicate flavor: " + flavor, meta);
+      }
+    }
+    return blocks;
+  }
+
+  /**
+   * Quantizes and writes a group's distinct vectors: one block per flavor, one record per group
+   * ordinal within each block. Wherever a {@code preQuantized} record with a matching encoding and
+   * flavor is available (i.e. the vector originates from a segment in this format), it is copied
+   * as-is; otherwise the raw vector is quantized. The block locations are written to {@code meta};
+   * non-FLOAT32 groups (stored raw only) record an empty list.
+   */
+  void writeGroup(
+      IndexOutput meta,
+      IndexOutput quantizedVectorData,
+      VectorEncoding groupEncoding,
+      int dimension,
+      int numVectors,
+      Set<Flavor> flavors,
+      FloatVectorSupplier vectors,
+      PreQuantizedSupplier preQuantized)
+      throws IOException {
+
+    if (groupEncoding != VectorEncoding.FLOAT32) {
+      writeEmptyGroup(meta);
+      return;
+    }
+
+    meta.writeVInt(flavors.size());
+    for (Flavor flavor : Flavor.values()) {
+      if (flavors.contains(flavor) == false) {
+        continue;
+      }
+      QuantizedBlock block =
+          writeFlavorBlock(
+              quantizedVectorData, flavor, dimension, numVectors, vectors, preQuantized);
+      meta.writeVInt(flavor.ordinal());
+      meta.writeVInt(block.encoding().getWireNumber());
+      meta.writeLong(block.quantizedDataOffset());
+      meta.writeLong(block.quantizedDataSize());
+    }
+  }
+
+  private QuantizedBlock writeFlavorBlock(
+      IndexOutput quantizedVectorData,
+      Flavor flavor,
+      int dimension,
+      int numVectors,
+      FloatVectorSupplier vectors,
+      PreQuantizedSupplier preQuantized)
+      throws IOException {
+
+    OptimizedScalarQuantizer quantizer = flavor.quantizer();
+    float[] zeroCentroid = new float[dimension];
+    float[] normalized = flavor.normalized() ? new float[dimension] : null;
+    byte[] scratch = new byte[encoding.getDiscreteDimensions(dimension)];
+    byte[] packed =
+        switch (encoding) {
+          case UNSIGNED_BYTE, SEVEN_BIT -> scratch;
+          case PACKED_NIBBLE, SINGLE_BIT_QUERY_NIBBLE, DIBIT_QUERY_NIBBLE ->
+              new byte[encoding.getDocPackedLength(scratch.length)];
+        };
+
+    long quantizedDataOffset = quantizedVectorData.alignFilePointer(Float.BYTES);
+    for (int ord = 0; ord < numVectors; ord++) {
+      // The quantized record is a pure function of the raw vector and the flavor: a record
+      // already computed by a source segment can be copied instead of re-quantizing.
+      PreQuantized pre = preQuantized == null ? null : preQuantized.get(ord);

Review Comment:
   Makes sense, done!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on PR #16506:
URL: https://github.com/apache/lucene/pull/16506#issuecomment-5499664061

   Opened #16594 for backport to 10.x!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on PR #16506:
URL: https://github.com/apache/lucene/pull/16506#issuecomment-5360071264

   > why we expect an increase in latency when de-duplicating, especially as the increase correlating with reduced quantized bit size
   
   I think this is a combination of two things: the fixed cost per-node for the indirection (off-heap `fieldOrdToGroupOrd` lookup) will be a larger _proportion_ of individual vector scoring with higher quantization (i.e. 1-bit vector computations are faster than 8-bit, so the additional latency from the indirection is a higher percentage increase) + more nodes being visited due to more approximate scores after loss of centering (also showing up as drop in recall).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on code in PR #16506:
URL: https://github.com/apache/lucene/pull/16506#discussion_r3846306734


##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupScalarQuantizedVectorsFormat.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import java.io.IOException;
+import org.apache.lucene.codecs.hnsw.FlatVectorsFormat;
+import org.apache.lucene.codecs.hnsw.FlatVectorsReader;
+import org.apache.lucene.codecs.hnsw.FlatVectorsWriter;
+import org.apache.lucene.index.SegmentReadState;
+import org.apache.lucene.index.SegmentWriteState;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * A scalar quantized version of {@link DedupFlatVectorsFormat} that stores each distinct vector
+ * once, in both raw and quantized form.
+ *
+ * <p>Quantization is <i>data-blind</i>: input vectors are assumed to be evenly distributed, i.e.
+ * centered on a zero vector (see {@link DedupQuantizer}). The quantized record of a vector is then
+ * a pure function of the raw vector and the {@link DedupQuantizer.Flavor} derived from the field's
+ * similarity function — independent of the other vectors in the segment — so quantized records
+ * de-duplicate like raw vectors: identical vectors across documents and fields of a group whose
+ * similarity functions map to the same flavor share one quantized record, resolved through the same
+ * {@code fieldOrdToGroupOrd} translation map. Records follow the {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsFormat} conventions per flavor,
+ * and are scored by the stock {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer}. Note the accuracy
+ * tradeoff relative to that format, which centers vectors on a per-field centroid before
+ * quantizing.
+ *
+ * <p>Only {@link org.apache.lucene.index.VectorEncoding#FLOAT32} vectors are quantized; BYTE and
+ * FLOAT16 vectors are stored raw only, identical to {@link DedupFlatVectorsFormat}.

Review Comment:
   Right, Lucene doesn't support quantizing FLOAT16 vectors today, we'll need to copy whatever the flat format does eventually.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on code in PR #16506:
URL: https://github.com/apache/lucene/pull/16506#discussion_r3846284249


##########
lucene/sandbox/src/test/org/apache/lucene/sandbox/codecs/dedup/TestDedupScalarQuantizedVectorsFormat.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import static org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsFormat.DEFAULT_BEAM_WIDTH;
+import static org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsFormat.DEFAULT_MAX_CONN;
+import static org.apache.lucene.index.VectorEncoding.BYTE;
+import static org.apache.lucene.index.VectorEncoding.FLOAT32;
+import static org.apache.lucene.index.VectorSimilarityFunction.COSINE;
+import static org.apache.lucene.index.VectorSimilarityFunction.DOT_PRODUCT;
+import static org.apache.lucene.index.VectorSimilarityFunction.EUCLIDEAN;
+import static org.apache.lucene.index.VectorSimilarityFunction.MAXIMUM_INNER_PRODUCT;
+import static org.hamcrest.Matchers.instanceOf;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.lucene.codecs.KnnVectorsReader;
+import org.apache.lucene.codecs.hnsw.FlatVectorsReader;
+import org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsReader;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.KnnByteVectorField;
+import org.apache.lucene.document.KnnFloatVectorField;
+import org.apache.lucene.document.NumericDocValuesField;
+import org.apache.lucene.index.ByteVectorValues;
+import org.apache.lucene.index.CodecReader;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.FloatVectorValues;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.KnnVectorValues;
+import org.apache.lucene.index.LeafReader;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.search.AcceptDocs;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.search.TopDocs;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.tests.util.LuceneTestCase;
+import org.apache.lucene.tests.util.TestUtil;
+import org.apache.lucene.util.VectorUtil;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * Tests that {@link DedupHnswScalarQuantizedVectorsFormat} stores each distinct vector once, in
+ * both raw and quantized form. General de-duplication behavior is covered by {@link
+ * TestDedupFlatVectorsFormat}; this test focuses on the quantized side.
+ */
+public class TestDedupScalarQuantizedVectorsFormat extends LuceneTestCase {

Review Comment:
   I'm not sure about how to reach the `#copy` method for those classes from public APIs, any suggestions to have a meaningful test? (or do you mean a simple one that creates a `new FieldOrdToGroupOrdArrayList` and expects an error on calling `#copy`)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on code in PR #16506:
URL: https://github.com/apache/lucene/pull/16506#discussion_r3846236674


##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupQuantizer.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import java.io.IOException;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.lucene.codecs.lucene104.OffHeapScalarQuantizedVectorValues;
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.VectorEncoding;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.VectorUtil;
+import org.apache.lucene.util.quantization.OptimizedScalarQuantizer;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * Write-side helper performing <i>data-blind</i> scalar quantization of de-duplicated vectors.
+ *
+ * <p>Quantization assumes input vectors are evenly distributed around the origin, i.e. the centroid
+ * is a zero vector. The quantized record of a vector (packed bytes, optimized intervals, an
+ * additional correction and the component sum) is then a pure function of the raw vector and the
+ * {@link Flavor} derived from the field's similarity function: independent of the other vectors in
+ * the segment. Identical vectors thus quantize identically and the quantized data de-duplicates
+ * along with the raw data, shared across all documents and fields of a group with the same flavor.
+ *
+ * <p>For the same reason, a record already computed by a source segment does not change on merge:
+ * when a distinct vector originates from a segment in this format (with the same encoding and
+ * flavor), its quantized record is <b>copied</b> instead of re-reading the raw vector and
+ * re-quantizing it.
+ *
+ * <p>Records follow the {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsFormat} conventions exactly
+ * (per flavor), so they are scored by the stock {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer}.
+ *
+ * @lucene.experimental
+ */
+record DedupQuantizer(ScalarEncoding encoding) {
+
+  /**
+   * How a vector is prepared and quantized, derived from the field's {@link
+   * VectorSimilarityFunction}. Fields of a group whose similarity functions map to the same flavor
+   * share one quantized record per distinct vector; a group stores one block per flavor in use.
+   */
+  enum Flavor {
+
+    /** Vectors quantized as-is; the additional correction holds the squared norm. */
+    EUCLIDEAN(VectorSimilarityFunction.EUCLIDEAN, false),
+
+    /**
+     * Vectors quantized as-is; the additional correction holds the dot product with the (zero)
+     * centroid, i.e. zero.
+     */
+    DOT_PRODUCT(VectorSimilarityFunction.DOT_PRODUCT, false),
+
+    /**
+     * Vectors are l2-normalized before quantization (for cosine similarity, scored as a dot
+     * product); the additional correction is zero as for {@link #DOT_PRODUCT}.
+     */
+    NORMALIZED(VectorSimilarityFunction.DOT_PRODUCT, true);
+
+    private final VectorSimilarityFunction quantizerFunction;
+    private final boolean normalized;
+
+    Flavor(VectorSimilarityFunction quantizerFunction, boolean normalized) {
+      this.quantizerFunction = quantizerFunction;
+      this.normalized = normalized;
+    }
+
+    static Flavor of(VectorSimilarityFunction function) {
+      return switch (function) {
+        case EUCLIDEAN -> EUCLIDEAN;
+        case DOT_PRODUCT, MAXIMUM_INNER_PRODUCT -> DOT_PRODUCT;
+        case COSINE -> NORMALIZED;
+      };
+    }
+
+    /**
+     * The quantizer producing this flavor's records against a zero centroid. NOTE: {@link
+     * #NORMALIZED} quantizes with {@link VectorSimilarityFunction#DOT_PRODUCT} (identical bytes and
+     * corrections, without the unit-centroid requirement of a cosine-configured quantizer).
+     */
+    OptimizedScalarQuantizer quantizer() {
+      return new OptimizedScalarQuantizer(quantizerFunction);
+    }
+
+    boolean normalized() {
+      return normalized;
+    }
+  }
+
+  /** Supplies the distinct (raw) float vector at a group ordinal. */
+  interface FloatVectorSupplier {
+    float[] get(int ord) throws IOException;
+  }
+
+  /**
+   * The already-quantized record of a distinct vector, held by a source segment at {@code ord} with
+   * the given flavor.
+   */
+  record PreQuantized(QuantizedByteVectorValues values, Flavor flavor, int ord) {}
+
+  /**
+   * Supplies the {@link PreQuantized} record of the distinct vector at a group ordinal, or {@code
+   * null} when its source segment does not hold one.
+   */
+  interface PreQuantizedSupplier {
+    PreQuantized get(int ord) throws IOException;
+  }
+
+  /** The encoding, offset and size of one flavor's quantized data block. */
+  record QuantizedBlock(
+      ScalarEncoding encoding, long quantizedDataOffset, long quantizedDataSize) {}
+
+  /** Writes an empty flavor-block list, for groups without quantized data. */
+  static void writeEmptyGroup(IndexOutput meta) throws IOException {
+    meta.writeVInt(0);
+  }
+
+  /** Reads the flavor-block list of a group, the counterpart of {@link #writeGroup}. */
+  static Map<Flavor, QuantizedBlock> readGroup(IndexInput meta) throws IOException {
+    int numFlavors = meta.readVInt();
+    Map<Flavor, QuantizedBlock> blocks = new EnumMap<>(Flavor.class);
+    for (int i = 0; i < numFlavors; i++) {
+      int flavorOrd = meta.readVInt();
+      if (flavorOrd < 0 || flavorOrd >= Flavor.values().length) {
+        throw new CorruptIndexException("Invalid flavor ordinal: " + flavorOrd, meta);
+      }
+      Flavor flavor = Flavor.values()[flavorOrd];
+      int wireNumber = meta.readVInt();
+      ScalarEncoding encoding =
+          ScalarEncoding.fromWireNumber(wireNumber)
+              .orElseThrow(
+                  () ->
+                      new CorruptIndexException(
+                          "Invalid scalar encoding wire number: " + wireNumber, meta));
+      long offset = meta.readLong();
+      long size = meta.readLong();
+      if (blocks.put(flavor, new QuantizedBlock(encoding, offset, size)) != null) {
+        throw new CorruptIndexException("Duplicate flavor: " + flavor, meta);
+      }
+    }
+    return blocks;
+  }
+
+  /**
+   * Quantizes and writes a group's distinct vectors: one block per flavor, one record per group
+   * ordinal within each block. Wherever a {@code preQuantized} record with a matching encoding and
+   * flavor is available (i.e. the vector originates from a segment in this format), it is copied
+   * as-is; otherwise the raw vector is quantized. The block locations are written to {@code meta};
+   * non-FLOAT32 groups (stored raw only) record an empty list.
+   */
+  void writeGroup(
+      IndexOutput meta,
+      IndexOutput quantizedVectorData,
+      VectorEncoding groupEncoding,
+      int dimension,
+      int numVectors,
+      Set<Flavor> flavors,
+      FloatVectorSupplier vectors,
+      PreQuantizedSupplier preQuantized)
+      throws IOException {
+
+    if (groupEncoding != VectorEncoding.FLOAT32) {
+      writeEmptyGroup(meta);
+      return;
+    }
+
+    meta.writeVInt(flavors.size());
+    for (Flavor flavor : Flavor.values()) {
+      if (flavors.contains(flavor) == false) {
+        continue;
+      }
+      QuantizedBlock block =
+          writeFlavorBlock(
+              quantizedVectorData, flavor, dimension, numVectors, vectors, preQuantized);
+      meta.writeVInt(flavor.ordinal());
+      meta.writeVInt(block.encoding().getWireNumber());
+      meta.writeLong(block.quantizedDataOffset());
+      meta.writeLong(block.quantizedDataSize());
+    }
+  }
+
+  private QuantizedBlock writeFlavorBlock(
+      IndexOutput quantizedVectorData,
+      Flavor flavor,
+      int dimension,
+      int numVectors,
+      FloatVectorSupplier vectors,
+      PreQuantizedSupplier preQuantized)
+      throws IOException {
+
+    OptimizedScalarQuantizer quantizer = flavor.quantizer();
+    float[] zeroCentroid = new float[dimension];
+    float[] normalized = flavor.normalized() ? new float[dimension] : null;
+    byte[] scratch = new byte[encoding.getDiscreteDimensions(dimension)];
+    byte[] packed =
+        switch (encoding) {
+          case UNSIGNED_BYTE, SEVEN_BIT -> scratch;
+          case PACKED_NIBBLE, SINGLE_BIT_QUERY_NIBBLE, DIBIT_QUERY_NIBBLE ->
+              new byte[encoding.getDocPackedLength(scratch.length)];
+        };
+
+    long quantizedDataOffset = quantizedVectorData.alignFilePointer(Float.BYTES);
+    for (int ord = 0; ord < numVectors; ord++) {
+      // The quantized record is a pure function of the raw vector and the flavor: a record
+      // already computed by a source segment can be copied instead of re-quantizing.
+      PreQuantized pre = preQuantized == null ? null : preQuantized.get(ord);
+      if (pre != null && pre.flavor() == flavor && pre.values().getScalarEncoding() == encoding) {
+        // NOTE: read the packed bytes before the corrective terms, which are then served from the
+        // record cached by the read of the packed bytes
+        writeRecord(
+            quantizedVectorData,
+            pre.values().vectorValue(pre.ord()),
+            pre.values().getCorrectiveTerms(pre.ord()));
+        continue;
+      }
+
+      float[] vector = vectors.get(ord);
+      if (flavor.normalized()) {
+        // normalize a copy: the source buffer is shared / owned by the group
+        System.arraycopy(vector, 0, normalized, 0, dimension);
+        VectorUtil.l2normalize(normalized);
+        vector = normalized;
+      }
+      // NOTE: scalarQuantize subtracts the centroid from the input in place, but with a zero
+      // centroid the values are unchanged, so shared / owned buffers are safe to pass directly.
+      OptimizedScalarQuantizer.QuantizationResult corrections =
+          quantizer.scalarQuantize(vector, scratch, encoding.getBits(), zeroCentroid);
+      switch (encoding) {
+        case PACKED_NIBBLE -> OffHeapScalarQuantizedVectorValues.packNibbles(scratch, packed);
+        case SINGLE_BIT_QUERY_NIBBLE -> OptimizedScalarQuantizer.packAsBinary(scratch, packed);
+        case DIBIT_QUERY_NIBBLE -> OptimizedScalarQuantizer.transposeDibit(scratch, packed);
+        case UNSIGNED_BYTE, SEVEN_BIT -> {}
+      }
+      writeRecord(quantizedVectorData, packed, corrections);
+    }
+    long quantizedDataSize = quantizedVectorData.getFilePointer() - quantizedDataOffset;
+
+    return new QuantizedBlock(encoding, quantizedDataOffset, quantizedDataSize);
+  }
+
+  private static void writeRecord(
+      IndexOutput quantizedVectorData,
+      byte[] packed,
+      OptimizedScalarQuantizer.QuantizationResult corrections)
+      throws IOException {
+    quantizedVectorData.writeBytes(packed, packed.length);
+    quantizedVectorData.writeInt(Float.floatToIntBits(corrections.lowerInterval()));

Review Comment:
   I think the [`additionalCorrection`](https://github.com/kaivalnp/lucene/blob/3710ac806ea6c7b0c26c5e9e88ad2e5e550d1a87/lucene/core/src/java/org/apache/lucene/util/quantization/OptimizedScalarQuantizer.java#L107) term is always zero for some flavors (`DOT_PRODUCT` and `NORMALIZED`, but not `EUCLIDEAN`).
   
   Writing the (no-op) correction factor has some benefits: like parity with the non de-duplicating format (allowing re-use of the scorer and vector values classes), and [aligning the quantized bytes to multiples of 4](https://github.com/kaivalnp/lucene/blob/3710ac806ea6c7b0c26c5e9e88ad2e5e550d1a87/lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104ScalarQuantizedVectorsWriter.java#L169) for optimal scoring performance.



##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupMergeContext.java:
##########
@@ -86,7 +88,18 @@ void addField(FieldInfo fieldInfo, MergeState mergeState) throws IOException {
             mergeState.segmentInfo.maxDoc()));
   }
 
-  void finish(IndexOutput meta, IndexOutput vectorData) throws IOException {
+  /**
+   * Merges each group's distinct vectors, followed by per-field metadata. When {@code quantizer} is
+   * non-null, a quantized copy of each FLOAT32 group is also written and its block location

Review Comment:
   Makes sense, changed `each FLOAT32 group` -> `each applicable group`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on PR #16506:
URL: https://github.com/apache/lucene/pull/16506#issuecomment-5261243742

   ### Benchmarks
   
   Cohere v3 vectors, 100K docs, 10K queries, maxConn=64, beamWidth=250, topK=100, fanout=100, force merged to 1 segment.
   
   `main`
   
   ```
   quantized  visited  index(s)  index_docs/s  merge(s)  force_merge(s)  index_size(MB)         filterStrategy  recall  latency(ms)  netCPU  avgCpuCount
          no    12962     19.93       5016.55     22.12           20.83          403.01  query-time-pre-filter   0.495        4.301   4.300        1.000
          no     6198     21.58       4633.71     20.02           35.27          603.03      index-time-filter   0.995        1.941   1.940        1.000
      8 bits    12994     19.85       5038.04     23.42           28.17          502.19  query-time-pre-filter   0.495        5.060   5.059        1.000
      8 bits     6202     21.93       4560.38     23.92           47.01          751.75      index-time-filter   0.985        2.411   2.410        1.000
      4 bits    13136     20.35       4915.21     13.21           20.34          453.40  query-time-pre-filter   0.495        2.946   2.945        1.000
      4 bits     6279     22.57       4431.25     12.31           33.11          678.62      index-time-filter   0.931        1.288   1.287        0.999
      2 bits    13667     20.51       4875.43     13.71           20.74          429.31  query-time-pre-filter   0.489        2.387   2.386        1.000
      2 bits     6587     22.28       4487.73     12.31           32.34          642.48      index-time-filter   0.842        0.960   0.960        0.999
      1 bits    15227     19.60       5101.26     11.41           17.39          418.30  query-time-pre-filter   0.452        1.996   1.995        1.000
      1 bits     7412     22.28       4488.13     10.21           28.86          625.90      index-time-filter   0.726        0.893   0.892        0.999
   ```
   
   This PR
   
   ```
   quantized  visited  index(s)  index_docs/s  merge(s)  force_merge(s)  index_size(MB)         filterStrategy  recall  latency(ms)  netCPU  avgCpuCount
          no    12956     19.80       5051.53     18.92           20.13          403.38  query-time-pre-filter   0.495        4.481   4.480        1.000
          no     6192     24.71       4047.11     25.32           30.99          409.01      index-time-filter   0.995        2.062   2.062        1.000
      8 bits    12965     19.87       5031.95     22.82           24.53          502.57  query-time-pre-filter   0.495        5.129   5.128        1.000
      8 bits     6206     25.26       3958.51     30.33           37.65          508.19      index-time-filter   0.985        2.372   2.371        0.999
      4 bits    13167     20.06       4985.04     11.31           16.64          453.79  query-time-pre-filter   0.495        2.911   2.910        1.000
      4 bits     6279     25.18       3971.56     19.82           24.26          459.44      index-time-filter   0.922        1.288   1.286        0.999
      2 bits    13765     20.51       4875.91     11.21           17.33          429.54  query-time-pre-filter   0.481        2.325   2.323        1.000
      2 bits     6688     25.47       3926.50     18.92           26.52          435.27      index-time-filter   0.809        1.038   1.037        0.999
      1 bits    15405     19.88       5029.68     10.41           15.19          418.55  query-time-pre-filter   0.434        2.139   2.138        1.000
      1 bits     7590     24.74       4041.87     16.51           24.12          424.78      index-time-filter   0.687        0.952   0.951        0.999
   ```
   
   Some observations:
   - `recall` is lower with the data-blind quantization, and drop increases with higher levels of quantization as expected (drop is <4% for this combination).
   - `latency` is higher with the de-duplication, and the increase as a percentage is higher with higher levels of quantization as expected (increase is <10% for this combination).
   - `index_size(MB)` drops with the de-duplicating vector format, and the index size is approximately the same b/w `query-time-pre-filter` and `index-time-filter` (differing in HNSW graph size).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on code in PR #16506:
URL: https://github.com/apache/lucene/pull/16506#discussion_r3846232793


##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupQuantizer.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import java.io.IOException;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.lucene.codecs.lucene104.OffHeapScalarQuantizedVectorValues;
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.VectorEncoding;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.VectorUtil;
+import org.apache.lucene.util.quantization.OptimizedScalarQuantizer;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * Write-side helper performing <i>data-blind</i> scalar quantization of de-duplicated vectors.
+ *
+ * <p>Quantization assumes input vectors are evenly distributed around the origin, i.e. the centroid
+ * is a zero vector. The quantized record of a vector (packed bytes, optimized intervals, an
+ * additional correction and the component sum) is then a pure function of the raw vector and the
+ * {@link Flavor} derived from the field's similarity function: independent of the other vectors in
+ * the segment. Identical vectors thus quantize identically and the quantized data de-duplicates
+ * along with the raw data, shared across all documents and fields of a group with the same flavor.
+ *
+ * <p>For the same reason, a record already computed by a source segment does not change on merge:
+ * when a distinct vector originates from a segment in this format (with the same encoding and
+ * flavor), its quantized record is <b>copied</b> instead of re-reading the raw vector and
+ * re-quantizing it.
+ *
+ * <p>Records follow the {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsFormat} conventions exactly
+ * (per flavor), so they are scored by the stock {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer}.
+ *
+ * @lucene.experimental
+ */
+record DedupQuantizer(ScalarEncoding encoding) {
+
+  /**
+   * How a vector is prepared and quantized, derived from the field's {@link
+   * VectorSimilarityFunction}. Fields of a group whose similarity functions map to the same flavor
+   * share one quantized record per distinct vector; a group stores one block per flavor in use.
+   */
+  enum Flavor {
+
+    /** Vectors quantized as-is; the additional correction holds the squared norm. */
+    EUCLIDEAN(VectorSimilarityFunction.EUCLIDEAN, false),
+
+    /**
+     * Vectors quantized as-is; the additional correction holds the dot product with the (zero)
+     * centroid, i.e. zero.
+     */
+    DOT_PRODUCT(VectorSimilarityFunction.DOT_PRODUCT, false),
+
+    /**
+     * Vectors are l2-normalized before quantization (for cosine similarity, scored as a dot
+     * product); the additional correction is zero as for {@link #DOT_PRODUCT}.
+     */
+    NORMALIZED(VectorSimilarityFunction.DOT_PRODUCT, true);
+
+    private final VectorSimilarityFunction quantizerFunction;
+    private final boolean normalized;
+
+    Flavor(VectorSimilarityFunction quantizerFunction, boolean normalized) {
+      this.quantizerFunction = quantizerFunction;
+      this.normalized = normalized;
+    }
+
+    static Flavor of(VectorSimilarityFunction function) {
+      return switch (function) {
+        case EUCLIDEAN -> EUCLIDEAN;
+        case DOT_PRODUCT, MAXIMUM_INNER_PRODUCT -> DOT_PRODUCT;
+        case COSINE -> NORMALIZED;
+      };
+    }
+
+    /**
+     * The quantizer producing this flavor's records against a zero centroid. NOTE: {@link
+     * #NORMALIZED} quantizes with {@link VectorSimilarityFunction#DOT_PRODUCT} (identical bytes and
+     * corrections, without the unit-centroid requirement of a cosine-configured quantizer).
+     */
+    OptimizedScalarQuantizer quantizer() {
+      return new OptimizedScalarQuantizer(quantizerFunction);
+    }
+
+    boolean normalized() {
+      return normalized;
+    }
+  }
+
+  /** Supplies the distinct (raw) float vector at a group ordinal. */
+  interface FloatVectorSupplier {
+    float[] get(int ord) throws IOException;
+  }
+
+  /**
+   * The already-quantized record of a distinct vector, held by a source segment at {@code ord} with
+   * the given flavor.
+   */
+  record PreQuantized(QuantizedByteVectorValues values, Flavor flavor, int ord) {}
+
+  /**
+   * Supplies the {@link PreQuantized} record of the distinct vector at a group ordinal, or {@code
+   * null} when its source segment does not hold one.
+   */
+  interface PreQuantizedSupplier {
+    PreQuantized get(int ord) throws IOException;
+  }
+
+  /** The encoding, offset and size of one flavor's quantized data block. */
+  record QuantizedBlock(
+      ScalarEncoding encoding, long quantizedDataOffset, long quantizedDataSize) {}
+
+  /** Writes an empty flavor-block list, for groups without quantized data. */
+  static void writeEmptyGroup(IndexOutput meta) throws IOException {
+    meta.writeVInt(0);
+  }
+
+  /** Reads the flavor-block list of a group, the counterpart of {@link #writeGroup}. */
+  static Map<Flavor, QuantizedBlock> readGroup(IndexInput meta) throws IOException {
+    int numFlavors = meta.readVInt();
+    Map<Flavor, QuantizedBlock> blocks = new EnumMap<>(Flavor.class);
+    for (int i = 0; i < numFlavors; i++) {
+      int flavorOrd = meta.readVInt();
+      if (flavorOrd < 0 || flavorOrd >= Flavor.values().length) {
+        throw new CorruptIndexException("Invalid flavor ordinal: " + flavorOrd, meta);
+      }
+      Flavor flavor = Flavor.values()[flavorOrd];
+      int wireNumber = meta.readVInt();
+      ScalarEncoding encoding =
+          ScalarEncoding.fromWireNumber(wireNumber)
+              .orElseThrow(
+                  () ->
+                      new CorruptIndexException(
+                          "Invalid scalar encoding wire number: " + wireNumber, meta));
+      long offset = meta.readLong();
+      long size = meta.readLong();
+      if (blocks.put(flavor, new QuantizedBlock(encoding, offset, size)) != null) {
+        throw new CorruptIndexException("Duplicate flavor: " + flavor, meta);
+      }
+    }
+    return blocks;
+  }
+
+  /**
+   * Quantizes and writes a group's distinct vectors: one block per flavor, one record per group
+   * ordinal within each block. Wherever a {@code preQuantized} record with a matching encoding and
+   * flavor is available (i.e. the vector originates from a segment in this format), it is copied
+   * as-is; otherwise the raw vector is quantized. The block locations are written to {@code meta};
+   * non-FLOAT32 groups (stored raw only) record an empty list.
+   */
+  void writeGroup(
+      IndexOutput meta,
+      IndexOutput quantizedVectorData,
+      VectorEncoding groupEncoding,
+      int dimension,
+      int numVectors,
+      Set<Flavor> flavors,
+      FloatVectorSupplier vectors,
+      PreQuantizedSupplier preQuantized)
+      throws IOException {
+
+    if (groupEncoding != VectorEncoding.FLOAT32) {
+      writeEmptyGroup(meta);
+      return;
+    }
+
+    meta.writeVInt(flavors.size());
+    for (Flavor flavor : Flavor.values()) {
+      if (flavors.contains(flavor) == false) {
+        continue;
+      }
+      QuantizedBlock block =
+          writeFlavorBlock(
+              quantizedVectorData, flavor, dimension, numVectors, vectors, preQuantized);
+      meta.writeVInt(flavor.ordinal());
+      meta.writeVInt(block.encoding().getWireNumber());
+      meta.writeLong(block.quantizedDataOffset());
+      meta.writeLong(block.quantizedDataSize());
+    }
+  }
+
+  private QuantizedBlock writeFlavorBlock(
+      IndexOutput quantizedVectorData,
+      Flavor flavor,
+      int dimension,
+      int numVectors,
+      FloatVectorSupplier vectors,
+      PreQuantizedSupplier preQuantized)
+      throws IOException {
+
+    OptimizedScalarQuantizer quantizer = flavor.quantizer();
+    float[] zeroCentroid = new float[dimension];
+    float[] normalized = flavor.normalized() ? new float[dimension] : null;
+    byte[] scratch = new byte[encoding.getDiscreteDimensions(dimension)];
+    byte[] packed =
+        switch (encoding) {
+          case UNSIGNED_BYTE, SEVEN_BIT -> scratch;
+          case PACKED_NIBBLE, SINGLE_BIT_QUERY_NIBBLE, DIBIT_QUERY_NIBBLE ->
+              new byte[encoding.getDocPackedLength(scratch.length)];
+        };
+
+    long quantizedDataOffset = quantizedVectorData.alignFilePointer(Float.BYTES);
+    for (int ord = 0; ord < numVectors; ord++) {
+      // The quantized record is a pure function of the raw vector and the flavor: a record
+      // already computed by a source segment can be copied instead of re-quantizing.
+      PreQuantized pre = preQuantized == null ? null : preQuantized.get(ord);
+      if (pre != null && pre.flavor() == flavor && pre.values().getScalarEncoding() == encoding) {
+        // NOTE: read the packed bytes before the corrective terms, which are then served from the
+        // record cached by the read of the packed bytes
+        writeRecord(
+            quantizedVectorData,
+            pre.values().vectorValue(pre.ord()),
+            pre.values().getCorrectiveTerms(pre.ord()));
+        continue;
+      }
+
+      float[] vector = vectors.get(ord);
+      if (flavor.normalized()) {
+        // normalize a copy: the source buffer is shared / owned by the group
+        System.arraycopy(vector, 0, normalized, 0, dimension);

Review Comment:
   For the flush case, `vector` [points to](https://github.com/kaivalnp/lucene/blob/3710ac806ea6c7b0c26c5e9e88ad2e5e550d1a87/lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupFlushContext.java#L158) the on-heap vector used to create HNSW graphs at the end -- so this is not safe to modify in-place.
   
   For the merge case, `vector` [points to](https://github.com/kaivalnp/lucene/blob/3710ac806ea6c7b0c26c5e9e88ad2e5e550d1a87/lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupMergeContext.java#L150) the output of `FloatVectorValues#vectorValue`, which is a [shared buffer](https://github.com/kaivalnp/lucene/blob/3710ac806ea6c7b0c26c5e9e88ad2e5e550d1a87/lucene/core/src/java/org/apache/lucene/codecs/lucene95/OffHeapFloatVectorValues.java#L77-L86) in case of an off-heap index -- this seems safe to modify on it's own, _provided_ that the same vector is not read again (`lastOrd` is cached).
   
   I went ahead with copying for safety (also, the same `float[]` is re-used instead of allocating a new one for each document).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on code in PR #16506:
URL: https://github.com/apache/lucene/pull/16506#discussion_r3847399916


##########
lucene/sandbox/src/test/org/apache/lucene/sandbox/codecs/dedup/TestDedupScalarQuantizedVectorsFormat.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import static org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsFormat.DEFAULT_BEAM_WIDTH;
+import static org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsFormat.DEFAULT_MAX_CONN;
+import static org.apache.lucene.index.VectorEncoding.BYTE;
+import static org.apache.lucene.index.VectorEncoding.FLOAT32;
+import static org.apache.lucene.index.VectorSimilarityFunction.COSINE;
+import static org.apache.lucene.index.VectorSimilarityFunction.DOT_PRODUCT;
+import static org.apache.lucene.index.VectorSimilarityFunction.EUCLIDEAN;
+import static org.apache.lucene.index.VectorSimilarityFunction.MAXIMUM_INNER_PRODUCT;
+import static org.hamcrest.Matchers.instanceOf;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.lucene.codecs.KnnVectorsReader;
+import org.apache.lucene.codecs.hnsw.FlatVectorsReader;
+import org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsReader;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.KnnByteVectorField;
+import org.apache.lucene.document.KnnFloatVectorField;
+import org.apache.lucene.document.NumericDocValuesField;
+import org.apache.lucene.index.ByteVectorValues;
+import org.apache.lucene.index.CodecReader;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.FloatVectorValues;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.KnnVectorValues;
+import org.apache.lucene.index.LeafReader;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.search.AcceptDocs;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.search.TopDocs;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.tests.util.LuceneTestCase;
+import org.apache.lucene.tests.util.TestUtil;
+import org.apache.lucene.util.VectorUtil;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * Tests that {@link DedupHnswScalarQuantizedVectorsFormat} stores each distinct vector once, in
+ * both raw and quantized form. General de-duplication behavior is covered by {@link
+ * TestDedupFlatVectorsFormat}; this test focuses on the quantized side.
+ */
+public class TestDedupScalarQuantizedVectorsFormat extends LuceneTestCase {

Review Comment:
   yeah, I was thinking of a simple unit test



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on code in PR #16506:
URL: https://github.com/apache/lucene/pull/16506#discussion_r3846237894


##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupScalarQuantizedVectorsFormat.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import java.io.IOException;
+import org.apache.lucene.codecs.hnsw.FlatVectorsFormat;
+import org.apache.lucene.codecs.hnsw.FlatVectorsReader;
+import org.apache.lucene.codecs.hnsw.FlatVectorsWriter;
+import org.apache.lucene.index.SegmentReadState;
+import org.apache.lucene.index.SegmentWriteState;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * A scalar quantized version of {@link DedupFlatVectorsFormat} that stores each distinct vector
+ * once, in both raw and quantized form.
+ *
+ * <p>Quantization is <i>data-blind</i>: input vectors are assumed to be evenly distributed, i.e.
+ * centered on a zero vector (see {@link DedupQuantizer}). The quantized record of a vector is then
+ * a pure function of the raw vector and the {@link DedupQuantizer.Flavor} derived from the field's
+ * similarity function — independent of the other vectors in the segment — so quantized records
+ * de-duplicate like raw vectors: identical vectors across documents and fields of a group whose
+ * similarity functions map to the same flavor share one quantized record, resolved through the same
+ * {@code fieldOrdToGroupOrd} translation map. Records follow the {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsFormat} conventions per flavor,
+ * and are scored by the stock {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer}. Note the accuracy
+ * tradeoff relative to that format, which centers vectors on a per-field centroid before
+ * quantizing.
+ *
+ * <p>Only {@link org.apache.lucene.index.VectorEncoding#FLOAT32} vectors are quantized; BYTE and
+ * FLOAT16 vectors are stored raw only, identical to {@link DedupFlatVectorsFormat}.

Review Comment:
   Right, I don't think we have the ability quantize `FLOAT16` vectors today -- we'll need to mirror the logic in the original scalar quantized format when it's added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on PR #16506:
URL: https://github.com/apache/lucene/pull/16506#issuecomment-5358046378

   So I understand why recall might drop a bit due to less-optimal quantization (no centering -- is that the only change?), but I'm less clear why we expect an increase in latency when de-duplicating, especially as the increase correlating with reduced quantized bit size.  I think I would expect a small increase, perhaps, due to the need for an indirection when looking up ordinals relative to a dense index, but this wouldn't vary with quantization.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on code in PR #16506:
URL: https://github.com/apache/lucene/pull/16506#discussion_r3846207794


##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupFlatVectorsScorer.java:
##########
@@ -37,112 +36,123 @@
  *
  * @lucene.experimental
  */
-final class DedupFlatVectorsScorer implements FlatVectorsScorer {
-  private static final FlatVectorsScorer SCORER =
+sealed class DedupFlatVectorsScorer implements FlatVectorsScorer
+    permits DedupScalarQuantizedVectorsScorer {
+
+  private static final FlatVectorsScorer FLAT_SCORER =
       FlatVectorScorerUtil.getLucene99FlatVectorsScorer();
 
+  private final FlatVectorsScorer scorer;

Review Comment:
   Makes sense, changed to `delegate`!



##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupFlatVectorsScorer.java:
##########
@@ -37,112 +36,123 @@
  *
  * @lucene.experimental
  */
-final class DedupFlatVectorsScorer implements FlatVectorsScorer {
-  private static final FlatVectorsScorer SCORER =
+sealed class DedupFlatVectorsScorer implements FlatVectorsScorer
+    permits DedupScalarQuantizedVectorsScorer {
+
+  private static final FlatVectorsScorer FLAT_SCORER =
       FlatVectorScorerUtil.getLucene99FlatVectorsScorer();
 
+  private final FlatVectorsScorer scorer;
+
+  DedupFlatVectorsScorer() {
+    this(FLAT_SCORER);
+  }
+
+  protected DedupFlatVectorsScorer(FlatVectorsScorer scorer) {
+    this.scorer = scorer;
+  }
+
+  /** Resolves the values to score; subclasses may unwrap composite values. */

Review Comment:
   Makes sense, changed!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on code in PR #16506:
URL: https://github.com/apache/lucene/pull/16506#discussion_r3846239756


##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupVectorValues.java:
##########
@@ -469,7 +475,7 @@ public int get(int ord) {
 
     @Override
     public FieldOrdToGroupOrd copy() {
-      return new FieldOrdToGroupOrdArrayList(fieldOrdToGroupOrd);
+      throw new UnsupportedOperationException("not meant for copying");

Review Comment:
   Yes, this is used during flush, and was never copied -- so I threw an exception instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on code in PR #16506:
URL: https://github.com/apache/lucene/pull/16506#discussion_r3847388866


##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupQuantizer.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import java.io.IOException;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.lucene.codecs.lucene104.OffHeapScalarQuantizedVectorValues;
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.VectorEncoding;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.VectorUtil;
+import org.apache.lucene.util.quantization.OptimizedScalarQuantizer;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * Write-side helper performing <i>data-blind</i> scalar quantization of de-duplicated vectors.
+ *
+ * <p>Quantization assumes input vectors are evenly distributed around the origin, i.e. the centroid
+ * is a zero vector. The quantized record of a vector (packed bytes, optimized intervals, an
+ * additional correction and the component sum) is then a pure function of the raw vector and the
+ * {@link Flavor} derived from the field's similarity function: independent of the other vectors in
+ * the segment. Identical vectors thus quantize identically and the quantized data de-duplicates
+ * along with the raw data, shared across all documents and fields of a group with the same flavor.
+ *
+ * <p>For the same reason, a record already computed by a source segment does not change on merge:
+ * when a distinct vector originates from a segment in this format (with the same encoding and
+ * flavor), its quantized record is <b>copied</b> instead of re-reading the raw vector and
+ * re-quantizing it.
+ *
+ * <p>Records follow the {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsFormat} conventions exactly
+ * (per flavor), so they are scored by the stock {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer}.
+ *
+ * @lucene.experimental
+ */
+record DedupQuantizer(ScalarEncoding encoding) {
+
+  /**
+   * How a vector is prepared and quantized, derived from the field's {@link
+   * VectorSimilarityFunction}. Fields of a group whose similarity functions map to the same flavor
+   * share one quantized record per distinct vector; a group stores one block per flavor in use.
+   */
+  enum Flavor {
+
+    /** Vectors quantized as-is; the additional correction holds the squared norm. */
+    EUCLIDEAN(VectorSimilarityFunction.EUCLIDEAN, false),
+
+    /**
+     * Vectors quantized as-is; the additional correction holds the dot product with the (zero)
+     * centroid, i.e. zero.
+     */
+    DOT_PRODUCT(VectorSimilarityFunction.DOT_PRODUCT, false),
+
+    /**
+     * Vectors are l2-normalized before quantization (for cosine similarity, scored as a dot
+     * product); the additional correction is zero as for {@link #DOT_PRODUCT}.
+     */
+    NORMALIZED(VectorSimilarityFunction.DOT_PRODUCT, true);
+
+    private final VectorSimilarityFunction quantizerFunction;
+    private final boolean normalized;
+
+    Flavor(VectorSimilarityFunction quantizerFunction, boolean normalized) {
+      this.quantizerFunction = quantizerFunction;
+      this.normalized = normalized;
+    }
+
+    static Flavor of(VectorSimilarityFunction function) {
+      return switch (function) {
+        case EUCLIDEAN -> EUCLIDEAN;
+        case DOT_PRODUCT, MAXIMUM_INNER_PRODUCT -> DOT_PRODUCT;
+        case COSINE -> NORMALIZED;
+      };
+    }
+
+    /**
+     * The quantizer producing this flavor's records against a zero centroid. NOTE: {@link
+     * #NORMALIZED} quantizes with {@link VectorSimilarityFunction#DOT_PRODUCT} (identical bytes and
+     * corrections, without the unit-centroid requirement of a cosine-configured quantizer).
+     */
+    OptimizedScalarQuantizer quantizer() {
+      return new OptimizedScalarQuantizer(quantizerFunction);
+    }
+
+    boolean normalized() {
+      return normalized;
+    }
+  }
+
+  /** Supplies the distinct (raw) float vector at a group ordinal. */
+  interface FloatVectorSupplier {
+    float[] get(int ord) throws IOException;
+  }
+
+  /**
+   * The already-quantized record of a distinct vector, held by a source segment at {@code ord} with
+   * the given flavor.
+   */
+  record PreQuantized(QuantizedByteVectorValues values, Flavor flavor, int ord) {}
+
+  /**
+   * Supplies the {@link PreQuantized} record of the distinct vector at a group ordinal, or {@code
+   * null} when its source segment does not hold one.
+   */
+  interface PreQuantizedSupplier {
+    PreQuantized get(int ord) throws IOException;
+  }
+
+  /** The encoding, offset and size of one flavor's quantized data block. */
+  record QuantizedBlock(
+      ScalarEncoding encoding, long quantizedDataOffset, long quantizedDataSize) {}
+
+  /** Writes an empty flavor-block list, for groups without quantized data. */
+  static void writeEmptyGroup(IndexOutput meta) throws IOException {
+    meta.writeVInt(0);
+  }
+
+  /** Reads the flavor-block list of a group, the counterpart of {@link #writeGroup}. */
+  static Map<Flavor, QuantizedBlock> readGroup(IndexInput meta) throws IOException {
+    int numFlavors = meta.readVInt();
+    Map<Flavor, QuantizedBlock> blocks = new EnumMap<>(Flavor.class);
+    for (int i = 0; i < numFlavors; i++) {
+      int flavorOrd = meta.readVInt();
+      if (flavorOrd < 0 || flavorOrd >= Flavor.values().length) {
+        throw new CorruptIndexException("Invalid flavor ordinal: " + flavorOrd, meta);
+      }
+      Flavor flavor = Flavor.values()[flavorOrd];
+      int wireNumber = meta.readVInt();
+      ScalarEncoding encoding =
+          ScalarEncoding.fromWireNumber(wireNumber)
+              .orElseThrow(
+                  () ->
+                      new CorruptIndexException(
+                          "Invalid scalar encoding wire number: " + wireNumber, meta));
+      long offset = meta.readLong();
+      long size = meta.readLong();
+      if (blocks.put(flavor, new QuantizedBlock(encoding, offset, size)) != null) {
+        throw new CorruptIndexException("Duplicate flavor: " + flavor, meta);
+      }
+    }
+    return blocks;
+  }
+
+  /**
+   * Quantizes and writes a group's distinct vectors: one block per flavor, one record per group
+   * ordinal within each block. Wherever a {@code preQuantized} record with a matching encoding and
+   * flavor is available (i.e. the vector originates from a segment in this format), it is copied
+   * as-is; otherwise the raw vector is quantized. The block locations are written to {@code meta};
+   * non-FLOAT32 groups (stored raw only) record an empty list.
+   */
+  void writeGroup(
+      IndexOutput meta,
+      IndexOutput quantizedVectorData,
+      VectorEncoding groupEncoding,
+      int dimension,
+      int numVectors,
+      Set<Flavor> flavors,
+      FloatVectorSupplier vectors,
+      PreQuantizedSupplier preQuantized)
+      throws IOException {
+
+    if (groupEncoding != VectorEncoding.FLOAT32) {
+      writeEmptyGroup(meta);
+      return;
+    }
+
+    meta.writeVInt(flavors.size());
+    for (Flavor flavor : Flavor.values()) {
+      if (flavors.contains(flavor) == false) {
+        continue;
+      }
+      QuantizedBlock block =
+          writeFlavorBlock(
+              quantizedVectorData, flavor, dimension, numVectors, vectors, preQuantized);
+      meta.writeVInt(flavor.ordinal());
+      meta.writeVInt(block.encoding().getWireNumber());
+      meta.writeLong(block.quantizedDataOffset());
+      meta.writeLong(block.quantizedDataSize());
+    }
+  }
+
+  private QuantizedBlock writeFlavorBlock(
+      IndexOutput quantizedVectorData,
+      Flavor flavor,
+      int dimension,
+      int numVectors,
+      FloatVectorSupplier vectors,
+      PreQuantizedSupplier preQuantized)
+      throws IOException {
+
+    OptimizedScalarQuantizer quantizer = flavor.quantizer();
+    float[] zeroCentroid = new float[dimension];
+    float[] normalized = flavor.normalized() ? new float[dimension] : null;
+    byte[] scratch = new byte[encoding.getDiscreteDimensions(dimension)];
+    byte[] packed =
+        switch (encoding) {
+          case UNSIGNED_BYTE, SEVEN_BIT -> scratch;
+          case PACKED_NIBBLE, SINGLE_BIT_QUERY_NIBBLE, DIBIT_QUERY_NIBBLE ->
+              new byte[encoding.getDocPackedLength(scratch.length)];
+        };
+
+    long quantizedDataOffset = quantizedVectorData.alignFilePointer(Float.BYTES);
+    for (int ord = 0; ord < numVectors; ord++) {
+      // The quantized record is a pure function of the raw vector and the flavor: a record
+      // already computed by a source segment can be copied instead of re-quantizing.
+      PreQuantized pre = preQuantized == null ? null : preQuantized.get(ord);
+      if (pre != null && pre.flavor() == flavor && pre.values().getScalarEncoding() == encoding) {
+        // NOTE: read the packed bytes before the corrective terms, which are then served from the
+        // record cached by the read of the packed bytes
+        writeRecord(
+            quantizedVectorData,
+            pre.values().vectorValue(pre.ord()),
+            pre.values().getCorrectiveTerms(pre.ord()));
+        continue;
+      }
+
+      float[] vector = vectors.get(ord);
+      if (flavor.normalized()) {
+        // normalize a copy: the source buffer is shared / owned by the group
+        System.arraycopy(vector, 0, normalized, 0, dimension);
+        VectorUtil.l2normalize(normalized);
+        vector = normalized;
+      }
+      // NOTE: scalarQuantize subtracts the centroid from the input in place, but with a zero
+      // centroid the values are unchanged, so shared / owned buffers are safe to pass directly.
+      OptimizedScalarQuantizer.QuantizationResult corrections =
+          quantizer.scalarQuantize(vector, scratch, encoding.getBits(), zeroCentroid);
+      switch (encoding) {
+        case PACKED_NIBBLE -> OffHeapScalarQuantizedVectorValues.packNibbles(scratch, packed);
+        case SINGLE_BIT_QUERY_NIBBLE -> OptimizedScalarQuantizer.packAsBinary(scratch, packed);
+        case DIBIT_QUERY_NIBBLE -> OptimizedScalarQuantizer.transposeDibit(scratch, packed);
+        case UNSIGNED_BYTE, SEVEN_BIT -> {}
+      }
+      writeRecord(quantizedVectorData, packed, corrections);
+    }
+    long quantizedDataSize = quantizedVectorData.getFilePointer() - quantizedDataOffset;
+
+    return new QuantizedBlock(encoding, quantizedDataOffset, quantizedDataSize);
+  }
+
+  private static void writeRecord(
+      IndexOutput quantizedVectorData,
+      byte[] packed,
+      OptimizedScalarQuantizer.QuantizationResult corrections)
+      throws IOException {
+    quantizedVectorData.writeBytes(packed, packed.length);
+    quantizedVectorData.writeInt(Float.floatToIntBits(corrections.lowerInterval()));

Review Comment:
   ok, the savings should be pretty small anyway



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on code in PR #16506:
URL: https://github.com/apache/lucene/pull/16506#discussion_r3846237894


##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupScalarQuantizedVectorsFormat.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import java.io.IOException;
+import org.apache.lucene.codecs.hnsw.FlatVectorsFormat;
+import org.apache.lucene.codecs.hnsw.FlatVectorsReader;
+import org.apache.lucene.codecs.hnsw.FlatVectorsWriter;
+import org.apache.lucene.index.SegmentReadState;
+import org.apache.lucene.index.SegmentWriteState;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * A scalar quantized version of {@link DedupFlatVectorsFormat} that stores each distinct vector
+ * once, in both raw and quantized form.
+ *
+ * <p>Quantization is <i>data-blind</i>: input vectors are assumed to be evenly distributed, i.e.
+ * centered on a zero vector (see {@link DedupQuantizer}). The quantized record of a vector is then
+ * a pure function of the raw vector and the {@link DedupQuantizer.Flavor} derived from the field's
+ * similarity function — independent of the other vectors in the segment — so quantized records
+ * de-duplicate like raw vectors: identical vectors across documents and fields of a group whose
+ * similarity functions map to the same flavor share one quantized record, resolved through the same
+ * {@code fieldOrdToGroupOrd} translation map. Records follow the {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsFormat} conventions per flavor,
+ * and are scored by the stock {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer}. Note the accuracy
+ * tradeoff relative to that format, which centers vectors on a per-field centroid before
+ * quantizing.
+ *
+ * <p>Only {@link org.apache.lucene.index.VectorEncoding#FLOAT32} vectors are quantized; BYTE and
+ * FLOAT16 vectors are stored raw only, identical to {@link DedupFlatVectorsFormat}.

Review Comment:
   Right, I don't think we have the ability quantize `FLOAT16` vectors today -- we'll need to mirror the logic in the original scalar quantized format when it's added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on code in PR #16506:
URL: https://github.com/apache/lucene/pull/16506#discussion_r3847420518


##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupQuantizer.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import java.io.IOException;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.lucene.codecs.lucene104.OffHeapScalarQuantizedVectorValues;
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.VectorEncoding;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.VectorUtil;
+import org.apache.lucene.util.quantization.OptimizedScalarQuantizer;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * Write-side helper performing <i>data-blind</i> scalar quantization of de-duplicated vectors.
+ *
+ * <p>Quantization assumes input vectors are evenly distributed around the origin, i.e. the centroid
+ * is a zero vector. The quantized record of a vector (packed bytes, optimized intervals, an
+ * additional correction and the component sum) is then a pure function of the raw vector and the
+ * {@link Flavor} derived from the field's similarity function: independent of the other vectors in
+ * the segment. Identical vectors thus quantize identically and the quantized data de-duplicates
+ * along with the raw data, shared across all documents and fields of a group with the same flavor.
+ *
+ * <p>For the same reason, a record already computed by a source segment does not change on merge:
+ * when a distinct vector originates from a segment in this format (with the same encoding and
+ * flavor), its quantized record is <b>copied</b> instead of re-reading the raw vector and
+ * re-quantizing it.
+ *
+ * <p>Records follow the {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsFormat} conventions exactly
+ * (per flavor), so they are scored by the stock {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer}.
+ *
+ * @lucene.experimental
+ */
+record DedupQuantizer(ScalarEncoding encoding) {
+
+  /**
+   * How a vector is prepared and quantized, derived from the field's {@link
+   * VectorSimilarityFunction}. Fields of a group whose similarity functions map to the same flavor
+   * share one quantized record per distinct vector; a group stores one block per flavor in use.
+   */
+  enum Flavor {
+
+    /** Vectors quantized as-is; the additional correction holds the squared norm. */
+    EUCLIDEAN(VectorSimilarityFunction.EUCLIDEAN, false),
+
+    /**
+     * Vectors quantized as-is; the additional correction holds the dot product with the (zero)
+     * centroid, i.e. zero.
+     */
+    DOT_PRODUCT(VectorSimilarityFunction.DOT_PRODUCT, false),
+
+    /**
+     * Vectors are l2-normalized before quantization (for cosine similarity, scored as a dot
+     * product); the additional correction is zero as for {@link #DOT_PRODUCT}.
+     */
+    NORMALIZED(VectorSimilarityFunction.DOT_PRODUCT, true);
+
+    private final VectorSimilarityFunction quantizerFunction;
+    private final boolean normalized;
+
+    Flavor(VectorSimilarityFunction quantizerFunction, boolean normalized) {
+      this.quantizerFunction = quantizerFunction;
+      this.normalized = normalized;
+    }
+
+    static Flavor of(VectorSimilarityFunction function) {
+      return switch (function) {
+        case EUCLIDEAN -> EUCLIDEAN;
+        case DOT_PRODUCT, MAXIMUM_INNER_PRODUCT -> DOT_PRODUCT;
+        case COSINE -> NORMALIZED;
+      };
+    }
+
+    /**
+     * The quantizer producing this flavor's records against a zero centroid. NOTE: {@link
+     * #NORMALIZED} quantizes with {@link VectorSimilarityFunction#DOT_PRODUCT} (identical bytes and
+     * corrections, without the unit-centroid requirement of a cosine-configured quantizer).
+     */
+    OptimizedScalarQuantizer quantizer() {
+      return new OptimizedScalarQuantizer(quantizerFunction);
+    }
+
+    boolean normalized() {
+      return normalized;
+    }
+  }
+
+  /** Supplies the distinct (raw) float vector at a group ordinal. */
+  interface FloatVectorSupplier {
+    float[] get(int ord) throws IOException;
+  }
+
+  /**
+   * The already-quantized record of a distinct vector, held by a source segment at {@code ord} with
+   * the given flavor.
+   */
+  record PreQuantized(QuantizedByteVectorValues values, Flavor flavor, int ord) {}
+
+  /**
+   * Supplies the {@link PreQuantized} record of the distinct vector at a group ordinal, or {@code
+   * null} when its source segment does not hold one.
+   */
+  interface PreQuantizedSupplier {
+    PreQuantized get(int ord) throws IOException;
+  }
+
+  /** The encoding, offset and size of one flavor's quantized data block. */
+  record QuantizedBlock(
+      ScalarEncoding encoding, long quantizedDataOffset, long quantizedDataSize) {}
+
+  /** Writes an empty flavor-block list, for groups without quantized data. */
+  static void writeEmptyGroup(IndexOutput meta) throws IOException {
+    meta.writeVInt(0);
+  }
+
+  /** Reads the flavor-block list of a group, the counterpart of {@link #writeGroup}. */
+  static Map<Flavor, QuantizedBlock> readGroup(IndexInput meta) throws IOException {
+    int numFlavors = meta.readVInt();
+    Map<Flavor, QuantizedBlock> blocks = new EnumMap<>(Flavor.class);
+    for (int i = 0; i < numFlavors; i++) {
+      int flavorOrd = meta.readVInt();
+      if (flavorOrd < 0 || flavorOrd >= Flavor.values().length) {
+        throw new CorruptIndexException("Invalid flavor ordinal: " + flavorOrd, meta);
+      }
+      Flavor flavor = Flavor.values()[flavorOrd];
+      int wireNumber = meta.readVInt();
+      ScalarEncoding encoding =
+          ScalarEncoding.fromWireNumber(wireNumber)
+              .orElseThrow(
+                  () ->
+                      new CorruptIndexException(
+                          "Invalid scalar encoding wire number: " + wireNumber, meta));
+      long offset = meta.readLong();
+      long size = meta.readLong();
+      if (blocks.put(flavor, new QuantizedBlock(encoding, offset, size)) != null) {
+        throw new CorruptIndexException("Duplicate flavor: " + flavor, meta);
+      }
+    }
+    return blocks;
+  }
+
+  /**
+   * Quantizes and writes a group's distinct vectors: one block per flavor, one record per group
+   * ordinal within each block. Wherever a {@code preQuantized} record with a matching encoding and
+   * flavor is available (i.e. the vector originates from a segment in this format), it is copied
+   * as-is; otherwise the raw vector is quantized. The block locations are written to {@code meta};
+   * non-FLOAT32 groups (stored raw only) record an empty list.
+   */
+  void writeGroup(
+      IndexOutput meta,
+      IndexOutput quantizedVectorData,
+      VectorEncoding groupEncoding,
+      int dimension,
+      int numVectors,
+      Set<Flavor> flavors,
+      FloatVectorSupplier vectors,
+      PreQuantizedSupplier preQuantized)
+      throws IOException {
+
+    if (groupEncoding != VectorEncoding.FLOAT32) {
+      writeEmptyGroup(meta);
+      return;
+    }
+
+    meta.writeVInt(flavors.size());
+    for (Flavor flavor : Flavor.values()) {
+      if (flavors.contains(flavor) == false) {
+        continue;
+      }
+      QuantizedBlock block =
+          writeFlavorBlock(
+              quantizedVectorData, flavor, dimension, numVectors, vectors, preQuantized);
+      meta.writeVInt(flavor.ordinal());
+      meta.writeVInt(block.encoding().getWireNumber());
+      meta.writeLong(block.quantizedDataOffset());
+      meta.writeLong(block.quantizedDataSize());
+    }
+  }
+
+  private QuantizedBlock writeFlavorBlock(
+      IndexOutput quantizedVectorData,
+      Flavor flavor,
+      int dimension,
+      int numVectors,
+      FloatVectorSupplier vectors,
+      PreQuantizedSupplier preQuantized)
+      throws IOException {
+
+    OptimizedScalarQuantizer quantizer = flavor.quantizer();
+    float[] zeroCentroid = new float[dimension];
+    float[] normalized = flavor.normalized() ? new float[dimension] : null;
+    byte[] scratch = new byte[encoding.getDiscreteDimensions(dimension)];
+    byte[] packed =
+        switch (encoding) {
+          case UNSIGNED_BYTE, SEVEN_BIT -> scratch;
+          case PACKED_NIBBLE, SINGLE_BIT_QUERY_NIBBLE, DIBIT_QUERY_NIBBLE ->
+              new byte[encoding.getDocPackedLength(scratch.length)];
+        };
+
+    long quantizedDataOffset = quantizedVectorData.alignFilePointer(Float.BYTES);
+    for (int ord = 0; ord < numVectors; ord++) {
+      // The quantized record is a pure function of the raw vector and the flavor: a record
+      // already computed by a source segment can be copied instead of re-quantizing.
+      PreQuantized pre = preQuantized == null ? null : preQuantized.get(ord);
+      if (pre != null && pre.flavor() == flavor && pre.values().getScalarEncoding() == encoding) {
+        // NOTE: read the packed bytes before the corrective terms, which are then served from the
+        // record cached by the read of the packed bytes
+        writeRecord(
+            quantizedVectorData,
+            pre.values().vectorValue(pre.ord()),
+            pre.values().getCorrectiveTerms(pre.ord()));
+        continue;
+      }
+
+      float[] vector = vectors.get(ord);
+      if (flavor.normalized()) {
+        // normalize a copy: the source buffer is shared / owned by the group
+        System.arraycopy(vector, 0, normalized, 0, dimension);

Review Comment:
   ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on code in PR #16506:
URL: https://github.com/apache/lucene/pull/16506#discussion_r3846209073


##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupFlatVectorsWriter.java:
##########
@@ -41,6 +35,10 @@
  * or merging existing segments (never both), delegating to {@link DedupFlushContext} or {@link
  * DedupMergeContext} accordingly.
  *
+ * <p>Also used by {@link DedupScalarQuantizedVectorsFormat} (with a non-null {@link
+ * DedupQuantizer}) to additionally write a quantized copy of each FLOAT32 group, into a separate

Review Comment:
   Makes sense, changed `each FLOAT32 group` -> `each applicable group` (considering that we may be able to quantize `FLOAT16` / `BYTE` vectors in the future)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on PR #16506:
URL: https://github.com/apache/lucene/pull/16506#issuecomment-5488530483

   FYI there was a bug in `luceneutil` that caused the baseline exact KNN results to be incorrect for the `query-time-pre-filter` cases (see benchmark https://github.com/apache/lucene/pull/16506#issuecomment-5261243742), leading to an artificially low recall. See https://github.com/mikemccand/luceneutil/pull/614 for the bug and fix. `index-time-filter` runs are completely unaffected, and everything apart from recall for `query-time-pre-filter` is unaffected.
   
   I re-ran a [full benchmark](https://gist.github.com/kaivalnp/90a5ca5d4424e116a0505bed4cf54707) for both `main` and this PR on 100K Cohere v3 vectors, 1024 dimensions, 10K queries, across all quantization levels and vector similarities (80 combinations each). TL;DR the recall for `query-time-pre-filter` is better than reported earlier, almost on par with `index-time-filter`. Everything else (performance benefit of `index-time-filter` over `query-time-pre-filter`, and indexing / search times of the flat and de-duplicating vector formats) is unchanged.
   
   I'll merge this soon!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp merged PR #16506:
URL: https://github.com/apache/lucene/pull/16506


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on code in PR #16506:
URL: https://github.com/apache/lucene/pull/16506#discussion_r3830262854


##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupFlatVectorsScorer.java:
##########
@@ -37,112 +36,123 @@
  *
  * @lucene.experimental
  */
-final class DedupFlatVectorsScorer implements FlatVectorsScorer {
-  private static final FlatVectorsScorer SCORER =
+sealed class DedupFlatVectorsScorer implements FlatVectorsScorer
+    permits DedupScalarQuantizedVectorsScorer {
+
+  private static final FlatVectorsScorer FLAT_SCORER =
       FlatVectorScorerUtil.getLucene99FlatVectorsScorer();
 
+  private final FlatVectorsScorer scorer;
+
+  DedupFlatVectorsScorer() {
+    this(FLAT_SCORER);
+  }
+
+  protected DedupFlatVectorsScorer(FlatVectorsScorer scorer) {
+    this.scorer = scorer;
+  }
+
+  /** Resolves the values to score; subclasses may unwrap composite values. */

Review Comment:
   Can we call it `unwrap`?  I was confused about what this was doing when I saw it used below



##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupFlatVectorsScorer.java:
##########
@@ -37,112 +36,123 @@
  *
  * @lucene.experimental
  */
-final class DedupFlatVectorsScorer implements FlatVectorsScorer {
-  private static final FlatVectorsScorer SCORER =
+sealed class DedupFlatVectorsScorer implements FlatVectorsScorer
+    permits DedupScalarQuantizedVectorsScorer {
+
+  private static final FlatVectorsScorer FLAT_SCORER =
       FlatVectorScorerUtil.getLucene99FlatVectorsScorer();
 
+  private final FlatVectorsScorer scorer;

Review Comment:
   since this is a scorer that delegates can we name this the conventional way as either `delegate` or `in`?



##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupFlatVectorsWriter.java:
##########
@@ -41,6 +35,10 @@
  * or merging existing segments (never both), delegating to {@link DedupFlushContext} or {@link
  * DedupMergeContext} accordingly.
  *
+ * <p>Also used by {@link DedupScalarQuantizedVectorsFormat} (with a non-null {@link
+ * DedupQuantizer}) to additionally write a quantized copy of each FLOAT32 group, into a separate

Review Comment:
   maybe in future they could be `FLOAT16`? Should we say each "full-precision" or "unqunatized" group?



##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupQuantizer.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import java.io.IOException;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.lucene.codecs.lucene104.OffHeapScalarQuantizedVectorValues;
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.VectorEncoding;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.VectorUtil;
+import org.apache.lucene.util.quantization.OptimizedScalarQuantizer;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * Write-side helper performing <i>data-blind</i> scalar quantization of de-duplicated vectors.
+ *
+ * <p>Quantization assumes input vectors are evenly distributed around the origin, i.e. the centroid
+ * is a zero vector. The quantized record of a vector (packed bytes, optimized intervals, an
+ * additional correction and the component sum) is then a pure function of the raw vector and the
+ * {@link Flavor} derived from the field's similarity function: independent of the other vectors in
+ * the segment. Identical vectors thus quantize identically and the quantized data de-duplicates
+ * along with the raw data, shared across all documents and fields of a group with the same flavor.
+ *
+ * <p>For the same reason, a record already computed by a source segment does not change on merge:
+ * when a distinct vector originates from a segment in this format (with the same encoding and
+ * flavor), its quantized record is <b>copied</b> instead of re-reading the raw vector and
+ * re-quantizing it.
+ *
+ * <p>Records follow the {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsFormat} conventions exactly
+ * (per flavor), so they are scored by the stock {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer}.
+ *
+ * @lucene.experimental
+ */
+record DedupQuantizer(ScalarEncoding encoding) {
+
+  /**
+   * How a vector is prepared and quantized, derived from the field's {@link
+   * VectorSimilarityFunction}. Fields of a group whose similarity functions map to the same flavor
+   * share one quantized record per distinct vector; a group stores one block per flavor in use.
+   */
+  enum Flavor {
+
+    /** Vectors quantized as-is; the additional correction holds the squared norm. */
+    EUCLIDEAN(VectorSimilarityFunction.EUCLIDEAN, false),
+
+    /**
+     * Vectors quantized as-is; the additional correction holds the dot product with the (zero)
+     * centroid, i.e. zero.
+     */
+    DOT_PRODUCT(VectorSimilarityFunction.DOT_PRODUCT, false),
+
+    /**
+     * Vectors are l2-normalized before quantization (for cosine similarity, scored as a dot
+     * product); the additional correction is zero as for {@link #DOT_PRODUCT}.
+     */
+    NORMALIZED(VectorSimilarityFunction.DOT_PRODUCT, true);
+
+    private final VectorSimilarityFunction quantizerFunction;
+    private final boolean normalized;
+
+    Flavor(VectorSimilarityFunction quantizerFunction, boolean normalized) {
+      this.quantizerFunction = quantizerFunction;
+      this.normalized = normalized;
+    }
+
+    static Flavor of(VectorSimilarityFunction function) {
+      return switch (function) {
+        case EUCLIDEAN -> EUCLIDEAN;
+        case DOT_PRODUCT, MAXIMUM_INNER_PRODUCT -> DOT_PRODUCT;
+        case COSINE -> NORMALIZED;
+      };
+    }
+
+    /**
+     * The quantizer producing this flavor's records against a zero centroid. NOTE: {@link
+     * #NORMALIZED} quantizes with {@link VectorSimilarityFunction#DOT_PRODUCT} (identical bytes and
+     * corrections, without the unit-centroid requirement of a cosine-configured quantizer).
+     */
+    OptimizedScalarQuantizer quantizer() {
+      return new OptimizedScalarQuantizer(quantizerFunction);
+    }
+
+    boolean normalized() {
+      return normalized;
+    }
+  }
+
+  /** Supplies the distinct (raw) float vector at a group ordinal. */
+  interface FloatVectorSupplier {

Review Comment:
   what's the visibility of this? If we later want to quantize Float16 (ie `short[]`) or even `byte[]` will we need a breaking change?



##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupQuantizer.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import java.io.IOException;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.lucene.codecs.lucene104.OffHeapScalarQuantizedVectorValues;
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.VectorEncoding;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.VectorUtil;
+import org.apache.lucene.util.quantization.OptimizedScalarQuantizer;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * Write-side helper performing <i>data-blind</i> scalar quantization of de-duplicated vectors.
+ *
+ * <p>Quantization assumes input vectors are evenly distributed around the origin, i.e. the centroid
+ * is a zero vector. The quantized record of a vector (packed bytes, optimized intervals, an
+ * additional correction and the component sum) is then a pure function of the raw vector and the
+ * {@link Flavor} derived from the field's similarity function: independent of the other vectors in
+ * the segment. Identical vectors thus quantize identically and the quantized data de-duplicates
+ * along with the raw data, shared across all documents and fields of a group with the same flavor.
+ *
+ * <p>For the same reason, a record already computed by a source segment does not change on merge:
+ * when a distinct vector originates from a segment in this format (with the same encoding and
+ * flavor), its quantized record is <b>copied</b> instead of re-reading the raw vector and
+ * re-quantizing it.
+ *
+ * <p>Records follow the {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsFormat} conventions exactly
+ * (per flavor), so they are scored by the stock {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer}.
+ *
+ * @lucene.experimental
+ */
+record DedupQuantizer(ScalarEncoding encoding) {
+
+  /**
+   * How a vector is prepared and quantized, derived from the field's {@link
+   * VectorSimilarityFunction}. Fields of a group whose similarity functions map to the same flavor
+   * share one quantized record per distinct vector; a group stores one block per flavor in use.
+   */
+  enum Flavor {
+
+    /** Vectors quantized as-is; the additional correction holds the squared norm. */
+    EUCLIDEAN(VectorSimilarityFunction.EUCLIDEAN, false),
+
+    /**
+     * Vectors quantized as-is; the additional correction holds the dot product with the (zero)
+     * centroid, i.e. zero.
+     */
+    DOT_PRODUCT(VectorSimilarityFunction.DOT_PRODUCT, false),
+
+    /**
+     * Vectors are l2-normalized before quantization (for cosine similarity, scored as a dot
+     * product); the additional correction is zero as for {@link #DOT_PRODUCT}.
+     */
+    NORMALIZED(VectorSimilarityFunction.DOT_PRODUCT, true);
+
+    private final VectorSimilarityFunction quantizerFunction;
+    private final boolean normalized;
+
+    Flavor(VectorSimilarityFunction quantizerFunction, boolean normalized) {
+      this.quantizerFunction = quantizerFunction;
+      this.normalized = normalized;
+    }
+
+    static Flavor of(VectorSimilarityFunction function) {
+      return switch (function) {
+        case EUCLIDEAN -> EUCLIDEAN;
+        case DOT_PRODUCT, MAXIMUM_INNER_PRODUCT -> DOT_PRODUCT;
+        case COSINE -> NORMALIZED;
+      };
+    }
+
+    /**
+     * The quantizer producing this flavor's records against a zero centroid. NOTE: {@link
+     * #NORMALIZED} quantizes with {@link VectorSimilarityFunction#DOT_PRODUCT} (identical bytes and
+     * corrections, without the unit-centroid requirement of a cosine-configured quantizer).
+     */
+    OptimizedScalarQuantizer quantizer() {
+      return new OptimizedScalarQuantizer(quantizerFunction);
+    }
+
+    boolean normalized() {
+      return normalized;
+    }
+  }
+
+  /** Supplies the distinct (raw) float vector at a group ordinal. */
+  interface FloatVectorSupplier {
+    float[] get(int ord) throws IOException;
+  }
+
+  /**
+   * The already-quantized record of a distinct vector, held by a source segment at {@code ord} with
+   * the given flavor.
+   */
+  record PreQuantized(QuantizedByteVectorValues values, Flavor flavor, int ord) {}
+
+  /**
+   * Supplies the {@link PreQuantized} record of the distinct vector at a group ordinal, or {@code
+   * null} when its source segment does not hold one.
+   */
+  interface PreQuantizedSupplier {
+    PreQuantized get(int ord) throws IOException;
+  }
+
+  /** The encoding, offset and size of one flavor's quantized data block. */
+  record QuantizedBlock(
+      ScalarEncoding encoding, long quantizedDataOffset, long quantizedDataSize) {}
+
+  /** Writes an empty flavor-block list, for groups without quantized data. */
+  static void writeEmptyGroup(IndexOutput meta) throws IOException {
+    meta.writeVInt(0);
+  }
+
+  /** Reads the flavor-block list of a group, the counterpart of {@link #writeGroup}. */
+  static Map<Flavor, QuantizedBlock> readGroup(IndexInput meta) throws IOException {
+    int numFlavors = meta.readVInt();
+    Map<Flavor, QuantizedBlock> blocks = new EnumMap<>(Flavor.class);
+    for (int i = 0; i < numFlavors; i++) {
+      int flavorOrd = meta.readVInt();
+      if (flavorOrd < 0 || flavorOrd >= Flavor.values().length) {
+        throw new CorruptIndexException("Invalid flavor ordinal: " + flavorOrd, meta);
+      }
+      Flavor flavor = Flavor.values()[flavorOrd];

Review Comment:
   Lucene doesn't generally rely on Enum ordinals assigned by the JVM and rather will explicitly define codes to store in the index, that can later be mapped to enums as needed.  This enables, for example, later removing support for an ordinal.  EG see the way VectorSimlarityFunction is handled in `Lucene94FieldInfosFormat`, or indeed the way `ScalarEncoding` is being deserialized here using a custom function (fromWireNumber)



##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupQuantizer.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import java.io.IOException;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.lucene.codecs.lucene104.OffHeapScalarQuantizedVectorValues;
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.VectorEncoding;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.VectorUtil;
+import org.apache.lucene.util.quantization.OptimizedScalarQuantizer;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * Write-side helper performing <i>data-blind</i> scalar quantization of de-duplicated vectors.
+ *
+ * <p>Quantization assumes input vectors are evenly distributed around the origin, i.e. the centroid
+ * is a zero vector. The quantized record of a vector (packed bytes, optimized intervals, an
+ * additional correction and the component sum) is then a pure function of the raw vector and the
+ * {@link Flavor} derived from the field's similarity function: independent of the other vectors in
+ * the segment. Identical vectors thus quantize identically and the quantized data de-duplicates
+ * along with the raw data, shared across all documents and fields of a group with the same flavor.
+ *
+ * <p>For the same reason, a record already computed by a source segment does not change on merge:
+ * when a distinct vector originates from a segment in this format (with the same encoding and
+ * flavor), its quantized record is <b>copied</b> instead of re-reading the raw vector and
+ * re-quantizing it.
+ *
+ * <p>Records follow the {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsFormat} conventions exactly
+ * (per flavor), so they are scored by the stock {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer}.
+ *
+ * @lucene.experimental
+ */
+record DedupQuantizer(ScalarEncoding encoding) {
+
+  /**
+   * How a vector is prepared and quantized, derived from the field's {@link
+   * VectorSimilarityFunction}. Fields of a group whose similarity functions map to the same flavor
+   * share one quantized record per distinct vector; a group stores one block per flavor in use.
+   */
+  enum Flavor {
+
+    /** Vectors quantized as-is; the additional correction holds the squared norm. */
+    EUCLIDEAN(VectorSimilarityFunction.EUCLIDEAN, false),
+
+    /**
+     * Vectors quantized as-is; the additional correction holds the dot product with the (zero)
+     * centroid, i.e. zero.
+     */
+    DOT_PRODUCT(VectorSimilarityFunction.DOT_PRODUCT, false),
+
+    /**
+     * Vectors are l2-normalized before quantization (for cosine similarity, scored as a dot
+     * product); the additional correction is zero as for {@link #DOT_PRODUCT}.
+     */
+    NORMALIZED(VectorSimilarityFunction.DOT_PRODUCT, true);
+
+    private final VectorSimilarityFunction quantizerFunction;
+    private final boolean normalized;
+
+    Flavor(VectorSimilarityFunction quantizerFunction, boolean normalized) {
+      this.quantizerFunction = quantizerFunction;
+      this.normalized = normalized;
+    }
+
+    static Flavor of(VectorSimilarityFunction function) {
+      return switch (function) {
+        case EUCLIDEAN -> EUCLIDEAN;
+        case DOT_PRODUCT, MAXIMUM_INNER_PRODUCT -> DOT_PRODUCT;
+        case COSINE -> NORMALIZED;
+      };
+    }
+
+    /**
+     * The quantizer producing this flavor's records against a zero centroid. NOTE: {@link
+     * #NORMALIZED} quantizes with {@link VectorSimilarityFunction#DOT_PRODUCT} (identical bytes and
+     * corrections, without the unit-centroid requirement of a cosine-configured quantizer).
+     */
+    OptimizedScalarQuantizer quantizer() {
+      return new OptimizedScalarQuantizer(quantizerFunction);
+    }
+
+    boolean normalized() {
+      return normalized;
+    }
+  }
+
+  /** Supplies the distinct (raw) float vector at a group ordinal. */
+  interface FloatVectorSupplier {
+    float[] get(int ord) throws IOException;
+  }
+
+  /**
+   * The already-quantized record of a distinct vector, held by a source segment at {@code ord} with
+   * the given flavor.
+   */
+  record PreQuantized(QuantizedByteVectorValues values, Flavor flavor, int ord) {}
+
+  /**
+   * Supplies the {@link PreQuantized} record of the distinct vector at a group ordinal, or {@code
+   * null} when its source segment does not hold one.
+   */
+  interface PreQuantizedSupplier {
+    PreQuantized get(int ord) throws IOException;
+  }
+
+  /** The encoding, offset and size of one flavor's quantized data block. */
+  record QuantizedBlock(
+      ScalarEncoding encoding, long quantizedDataOffset, long quantizedDataSize) {}
+
+  /** Writes an empty flavor-block list, for groups without quantized data. */
+  static void writeEmptyGroup(IndexOutput meta) throws IOException {
+    meta.writeVInt(0);
+  }
+
+  /** Reads the flavor-block list of a group, the counterpart of {@link #writeGroup}. */
+  static Map<Flavor, QuantizedBlock> readGroup(IndexInput meta) throws IOException {
+    int numFlavors = meta.readVInt();
+    Map<Flavor, QuantizedBlock> blocks = new EnumMap<>(Flavor.class);
+    for (int i = 0; i < numFlavors; i++) {
+      int flavorOrd = meta.readVInt();
+      if (flavorOrd < 0 || flavorOrd >= Flavor.values().length) {
+        throw new CorruptIndexException("Invalid flavor ordinal: " + flavorOrd, meta);
+      }
+      Flavor flavor = Flavor.values()[flavorOrd];
+      int wireNumber = meta.readVInt();
+      ScalarEncoding encoding =
+          ScalarEncoding.fromWireNumber(wireNumber)
+              .orElseThrow(
+                  () ->
+                      new CorruptIndexException(
+                          "Invalid scalar encoding wire number: " + wireNumber, meta));
+      long offset = meta.readLong();
+      long size = meta.readLong();
+      if (blocks.put(flavor, new QuantizedBlock(encoding, offset, size)) != null) {
+        throw new CorruptIndexException("Duplicate flavor: " + flavor, meta);
+      }
+    }
+    return blocks;
+  }
+
+  /**
+   * Quantizes and writes a group's distinct vectors: one block per flavor, one record per group
+   * ordinal within each block. Wherever a {@code preQuantized} record with a matching encoding and
+   * flavor is available (i.e. the vector originates from a segment in this format), it is copied
+   * as-is; otherwise the raw vector is quantized. The block locations are written to {@code meta};
+   * non-FLOAT32 groups (stored raw only) record an empty list.
+   */
+  void writeGroup(
+      IndexOutput meta,
+      IndexOutput quantizedVectorData,
+      VectorEncoding groupEncoding,
+      int dimension,
+      int numVectors,
+      Set<Flavor> flavors,
+      FloatVectorSupplier vectors,
+      PreQuantizedSupplier preQuantized)
+      throws IOException {
+
+    if (groupEncoding != VectorEncoding.FLOAT32) {
+      writeEmptyGroup(meta);
+      return;
+    }
+
+    meta.writeVInt(flavors.size());
+    for (Flavor flavor : Flavor.values()) {
+      if (flavors.contains(flavor) == false) {
+        continue;
+      }
+      QuantizedBlock block =
+          writeFlavorBlock(
+              quantizedVectorData, flavor, dimension, numVectors, vectors, preQuantized);
+      meta.writeVInt(flavor.ordinal());
+      meta.writeVInt(block.encoding().getWireNumber());
+      meta.writeLong(block.quantizedDataOffset());
+      meta.writeLong(block.quantizedDataSize());
+    }
+  }
+
+  private QuantizedBlock writeFlavorBlock(
+      IndexOutput quantizedVectorData,
+      Flavor flavor,
+      int dimension,
+      int numVectors,
+      FloatVectorSupplier vectors,
+      PreQuantizedSupplier preQuantized)
+      throws IOException {
+
+    OptimizedScalarQuantizer quantizer = flavor.quantizer();
+    float[] zeroCentroid = new float[dimension];
+    float[] normalized = flavor.normalized() ? new float[dimension] : null;
+    byte[] scratch = new byte[encoding.getDiscreteDimensions(dimension)];
+    byte[] packed =
+        switch (encoding) {
+          case UNSIGNED_BYTE, SEVEN_BIT -> scratch;
+          case PACKED_NIBBLE, SINGLE_BIT_QUERY_NIBBLE, DIBIT_QUERY_NIBBLE ->
+              new byte[encoding.getDocPackedLength(scratch.length)];
+        };
+
+    long quantizedDataOffset = quantizedVectorData.alignFilePointer(Float.BYTES);
+    for (int ord = 0; ord < numVectors; ord++) {
+      // The quantized record is a pure function of the raw vector and the flavor: a record
+      // already computed by a source segment can be copied instead of re-quantizing.
+      PreQuantized pre = preQuantized == null ? null : preQuantized.get(ord);

Review Comment:
   I feel like this would read clearer with an explicit `if (preQuantized == null) {` wrapping the following `if`. We don't use `pre` anywhere below, so the only value of assigning null to it is to skip the block, which we could do in a more idiomatic way with an if statement.



##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupMergeContext.java:
##########
@@ -86,7 +88,18 @@ void addField(FieldInfo fieldInfo, MergeState mergeState) throws IOException {
             mergeState.segmentInfo.maxDoc()));
   }
 
-  void finish(IndexOutput meta, IndexOutput vectorData) throws IOException {
+  /**
+   * Merges each group's distinct vectors, followed by per-field metadata. When {@code quantizer} is
+   * non-null, a quantized copy of each FLOAT32 group is also written and its block location

Review Comment:
   each "full precision" group?



##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupScalarQuantizedVectorsFormat.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import java.io.IOException;
+import org.apache.lucene.codecs.hnsw.FlatVectorsFormat;
+import org.apache.lucene.codecs.hnsw.FlatVectorsReader;
+import org.apache.lucene.codecs.hnsw.FlatVectorsWriter;
+import org.apache.lucene.index.SegmentReadState;
+import org.apache.lucene.index.SegmentWriteState;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * A scalar quantized version of {@link DedupFlatVectorsFormat} that stores each distinct vector
+ * once, in both raw and quantized form.
+ *
+ * <p>Quantization is <i>data-blind</i>: input vectors are assumed to be evenly distributed, i.e.
+ * centered on a zero vector (see {@link DedupQuantizer}). The quantized record of a vector is then
+ * a pure function of the raw vector and the {@link DedupQuantizer.Flavor} derived from the field's
+ * similarity function — independent of the other vectors in the segment — so quantized records
+ * de-duplicate like raw vectors: identical vectors across documents and fields of a group whose
+ * similarity functions map to the same flavor share one quantized record, resolved through the same
+ * {@code fieldOrdToGroupOrd} translation map. Records follow the {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsFormat} conventions per flavor,
+ * and are scored by the stock {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer}. Note the accuracy
+ * tradeoff relative to that format, which centers vectors on a per-field centroid before
+ * quantizing.
+ *
+ * <p>Only {@link org.apache.lucene.index.VectorEncoding#FLOAT32} vectors are quantized; BYTE and
+ * FLOAT16 vectors are stored raw only, identical to {@link DedupFlatVectorsFormat}.

Review Comment:
   ah, okay, it's by design. But I think we would eventually want to be able to quantize FLOAT16 too?



##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupQuantizer.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import java.io.IOException;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.lucene.codecs.lucene104.OffHeapScalarQuantizedVectorValues;
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.VectorEncoding;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.VectorUtil;
+import org.apache.lucene.util.quantization.OptimizedScalarQuantizer;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * Write-side helper performing <i>data-blind</i> scalar quantization of de-duplicated vectors.
+ *
+ * <p>Quantization assumes input vectors are evenly distributed around the origin, i.e. the centroid
+ * is a zero vector. The quantized record of a vector (packed bytes, optimized intervals, an
+ * additional correction and the component sum) is then a pure function of the raw vector and the
+ * {@link Flavor} derived from the field's similarity function: independent of the other vectors in
+ * the segment. Identical vectors thus quantize identically and the quantized data de-duplicates
+ * along with the raw data, shared across all documents and fields of a group with the same flavor.
+ *
+ * <p>For the same reason, a record already computed by a source segment does not change on merge:
+ * when a distinct vector originates from a segment in this format (with the same encoding and
+ * flavor), its quantized record is <b>copied</b> instead of re-reading the raw vector and
+ * re-quantizing it.
+ *
+ * <p>Records follow the {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsFormat} conventions exactly
+ * (per flavor), so they are scored by the stock {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer}.
+ *
+ * @lucene.experimental
+ */
+record DedupQuantizer(ScalarEncoding encoding) {
+
+  /**
+   * How a vector is prepared and quantized, derived from the field's {@link
+   * VectorSimilarityFunction}. Fields of a group whose similarity functions map to the same flavor
+   * share one quantized record per distinct vector; a group stores one block per flavor in use.
+   */
+  enum Flavor {
+
+    /** Vectors quantized as-is; the additional correction holds the squared norm. */
+    EUCLIDEAN(VectorSimilarityFunction.EUCLIDEAN, false),
+
+    /**
+     * Vectors quantized as-is; the additional correction holds the dot product with the (zero)
+     * centroid, i.e. zero.
+     */
+    DOT_PRODUCT(VectorSimilarityFunction.DOT_PRODUCT, false),
+
+    /**
+     * Vectors are l2-normalized before quantization (for cosine similarity, scored as a dot
+     * product); the additional correction is zero as for {@link #DOT_PRODUCT}.
+     */
+    NORMALIZED(VectorSimilarityFunction.DOT_PRODUCT, true);
+
+    private final VectorSimilarityFunction quantizerFunction;
+    private final boolean normalized;
+
+    Flavor(VectorSimilarityFunction quantizerFunction, boolean normalized) {
+      this.quantizerFunction = quantizerFunction;
+      this.normalized = normalized;
+    }
+
+    static Flavor of(VectorSimilarityFunction function) {
+      return switch (function) {
+        case EUCLIDEAN -> EUCLIDEAN;
+        case DOT_PRODUCT, MAXIMUM_INNER_PRODUCT -> DOT_PRODUCT;
+        case COSINE -> NORMALIZED;
+      };
+    }
+
+    /**
+     * The quantizer producing this flavor's records against a zero centroid. NOTE: {@link
+     * #NORMALIZED} quantizes with {@link VectorSimilarityFunction#DOT_PRODUCT} (identical bytes and
+     * corrections, without the unit-centroid requirement of a cosine-configured quantizer).
+     */
+    OptimizedScalarQuantizer quantizer() {
+      return new OptimizedScalarQuantizer(quantizerFunction);
+    }
+
+    boolean normalized() {
+      return normalized;
+    }
+  }
+
+  /** Supplies the distinct (raw) float vector at a group ordinal. */
+  interface FloatVectorSupplier {
+    float[] get(int ord) throws IOException;
+  }
+
+  /**
+   * The already-quantized record of a distinct vector, held by a source segment at {@code ord} with
+   * the given flavor.
+   */
+  record PreQuantized(QuantizedByteVectorValues values, Flavor flavor, int ord) {}
+
+  /**
+   * Supplies the {@link PreQuantized} record of the distinct vector at a group ordinal, or {@code
+   * null} when its source segment does not hold one.
+   */
+  interface PreQuantizedSupplier {
+    PreQuantized get(int ord) throws IOException;
+  }
+
+  /** The encoding, offset and size of one flavor's quantized data block. */
+  record QuantizedBlock(
+      ScalarEncoding encoding, long quantizedDataOffset, long quantizedDataSize) {}
+
+  /** Writes an empty flavor-block list, for groups without quantized data. */
+  static void writeEmptyGroup(IndexOutput meta) throws IOException {
+    meta.writeVInt(0);
+  }
+
+  /** Reads the flavor-block list of a group, the counterpart of {@link #writeGroup}. */
+  static Map<Flavor, QuantizedBlock> readGroup(IndexInput meta) throws IOException {
+    int numFlavors = meta.readVInt();
+    Map<Flavor, QuantizedBlock> blocks = new EnumMap<>(Flavor.class);
+    for (int i = 0; i < numFlavors; i++) {
+      int flavorOrd = meta.readVInt();
+      if (flavorOrd < 0 || flavorOrd >= Flavor.values().length) {
+        throw new CorruptIndexException("Invalid flavor ordinal: " + flavorOrd, meta);
+      }
+      Flavor flavor = Flavor.values()[flavorOrd];
+      int wireNumber = meta.readVInt();
+      ScalarEncoding encoding =
+          ScalarEncoding.fromWireNumber(wireNumber)
+              .orElseThrow(
+                  () ->
+                      new CorruptIndexException(
+                          "Invalid scalar encoding wire number: " + wireNumber, meta));
+      long offset = meta.readLong();
+      long size = meta.readLong();
+      if (blocks.put(flavor, new QuantizedBlock(encoding, offset, size)) != null) {
+        throw new CorruptIndexException("Duplicate flavor: " + flavor, meta);
+      }
+    }
+    return blocks;
+  }
+
+  /**
+   * Quantizes and writes a group's distinct vectors: one block per flavor, one record per group
+   * ordinal within each block. Wherever a {@code preQuantized} record with a matching encoding and
+   * flavor is available (i.e. the vector originates from a segment in this format), it is copied
+   * as-is; otherwise the raw vector is quantized. The block locations are written to {@code meta};
+   * non-FLOAT32 groups (stored raw only) record an empty list.
+   */
+  void writeGroup(
+      IndexOutput meta,
+      IndexOutput quantizedVectorData,
+      VectorEncoding groupEncoding,
+      int dimension,
+      int numVectors,
+      Set<Flavor> flavors,
+      FloatVectorSupplier vectors,
+      PreQuantizedSupplier preQuantized)
+      throws IOException {
+
+    if (groupEncoding != VectorEncoding.FLOAT32) {
+      writeEmptyGroup(meta);
+      return;
+    }
+
+    meta.writeVInt(flavors.size());
+    for (Flavor flavor : Flavor.values()) {
+      if (flavors.contains(flavor) == false) {
+        continue;
+      }
+      QuantizedBlock block =
+          writeFlavorBlock(
+              quantizedVectorData, flavor, dimension, numVectors, vectors, preQuantized);
+      meta.writeVInt(flavor.ordinal());
+      meta.writeVInt(block.encoding().getWireNumber());
+      meta.writeLong(block.quantizedDataOffset());
+      meta.writeLong(block.quantizedDataSize());
+    }
+  }
+
+  private QuantizedBlock writeFlavorBlock(
+      IndexOutput quantizedVectorData,
+      Flavor flavor,
+      int dimension,
+      int numVectors,
+      FloatVectorSupplier vectors,
+      PreQuantizedSupplier preQuantized)
+      throws IOException {
+
+    OptimizedScalarQuantizer quantizer = flavor.quantizer();
+    float[] zeroCentroid = new float[dimension];
+    float[] normalized = flavor.normalized() ? new float[dimension] : null;
+    byte[] scratch = new byte[encoding.getDiscreteDimensions(dimension)];
+    byte[] packed =
+        switch (encoding) {
+          case UNSIGNED_BYTE, SEVEN_BIT -> scratch;
+          case PACKED_NIBBLE, SINGLE_BIT_QUERY_NIBBLE, DIBIT_QUERY_NIBBLE ->
+              new byte[encoding.getDocPackedLength(scratch.length)];
+        };
+
+    long quantizedDataOffset = quantizedVectorData.alignFilePointer(Float.BYTES);
+    for (int ord = 0; ord < numVectors; ord++) {
+      // The quantized record is a pure function of the raw vector and the flavor: a record
+      // already computed by a source segment can be copied instead of re-quantizing.
+      PreQuantized pre = preQuantized == null ? null : preQuantized.get(ord);
+      if (pre != null && pre.flavor() == flavor && pre.values().getScalarEncoding() == encoding) {
+        // NOTE: read the packed bytes before the corrective terms, which are then served from the
+        // record cached by the read of the packed bytes
+        writeRecord(
+            quantizedVectorData,
+            pre.values().vectorValue(pre.ord()),
+            pre.values().getCorrectiveTerms(pre.ord()));
+        continue;
+      }
+
+      float[] vector = vectors.get(ord);
+      if (flavor.normalized()) {
+        // normalize a copy: the source buffer is shared / owned by the group
+        System.arraycopy(vector, 0, normalized, 0, dimension);
+        VectorUtil.l2normalize(normalized);
+        vector = normalized;
+      }
+      // NOTE: scalarQuantize subtracts the centroid from the input in place, but with a zero
+      // centroid the values are unchanged, so shared / owned buffers are safe to pass directly.
+      OptimizedScalarQuantizer.QuantizationResult corrections =
+          quantizer.scalarQuantize(vector, scratch, encoding.getBits(), zeroCentroid);
+      switch (encoding) {
+        case PACKED_NIBBLE -> OffHeapScalarQuantizedVectorValues.packNibbles(scratch, packed);
+        case SINGLE_BIT_QUERY_NIBBLE -> OptimizedScalarQuantizer.packAsBinary(scratch, packed);
+        case DIBIT_QUERY_NIBBLE -> OptimizedScalarQuantizer.transposeDibit(scratch, packed);
+        case UNSIGNED_BYTE, SEVEN_BIT -> {}
+      }
+      writeRecord(quantizedVectorData, packed, corrections);
+    }
+    long quantizedDataSize = quantizedVectorData.getFilePointer() - quantizedDataOffset;
+
+    return new QuantizedBlock(encoding, quantizedDataOffset, quantizedDataSize);
+  }
+
+  private static void writeRecord(
+      IndexOutput quantizedVectorData,
+      byte[] packed,
+      OptimizedScalarQuantizer.QuantizationResult corrections)
+      throws IOException {
+    quantizedVectorData.writeBytes(packed, packed.length);
+    quantizedVectorData.writeInt(Float.floatToIntBits(corrections.lowerInterval()));

Review Comment:
   for flavors where the corrections are uniformly zero, can we skip writing them?



##########
lucene/sandbox/src/test/org/apache/lucene/sandbox/codecs/dedup/TestDedupScalarQuantizedVectorsFormat.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import static org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsFormat.DEFAULT_BEAM_WIDTH;
+import static org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsFormat.DEFAULT_MAX_CONN;
+import static org.apache.lucene.index.VectorEncoding.BYTE;
+import static org.apache.lucene.index.VectorEncoding.FLOAT32;
+import static org.apache.lucene.index.VectorSimilarityFunction.COSINE;
+import static org.apache.lucene.index.VectorSimilarityFunction.DOT_PRODUCT;
+import static org.apache.lucene.index.VectorSimilarityFunction.EUCLIDEAN;
+import static org.apache.lucene.index.VectorSimilarityFunction.MAXIMUM_INNER_PRODUCT;
+import static org.hamcrest.Matchers.instanceOf;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.lucene.codecs.KnnVectorsReader;
+import org.apache.lucene.codecs.hnsw.FlatVectorsReader;
+import org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsReader;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.KnnByteVectorField;
+import org.apache.lucene.document.KnnFloatVectorField;
+import org.apache.lucene.document.NumericDocValuesField;
+import org.apache.lucene.index.ByteVectorValues;
+import org.apache.lucene.index.CodecReader;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.FloatVectorValues;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.KnnVectorValues;
+import org.apache.lucene.index.LeafReader;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.search.AcceptDocs;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.search.TopDocs;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.tests.util.LuceneTestCase;
+import org.apache.lucene.tests.util.TestUtil;
+import org.apache.lucene.util.VectorUtil;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * Tests that {@link DedupHnswScalarQuantizedVectorsFormat} stores each distinct vector once, in
+ * both raw and quantized form. General de-duplication behavior is covered by {@link
+ * TestDedupFlatVectorsFormat}; this test focuses on the quantized side.
+ */
+public class TestDedupScalarQuantizedVectorsFormat extends LuceneTestCase {

Review Comment:
   since we explicitly forbade copy() on some the vector values, let's add a test ensuring that those methods throw Unsupported exception 



##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupQuantizer.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import java.io.IOException;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.Set;
+import org.apache.lucene.codecs.lucene104.OffHeapScalarQuantizedVectorValues;
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.VectorEncoding;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.util.VectorUtil;
+import org.apache.lucene.util.quantization.OptimizedScalarQuantizer;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * Write-side helper performing <i>data-blind</i> scalar quantization of de-duplicated vectors.
+ *
+ * <p>Quantization assumes input vectors are evenly distributed around the origin, i.e. the centroid
+ * is a zero vector. The quantized record of a vector (packed bytes, optimized intervals, an
+ * additional correction and the component sum) is then a pure function of the raw vector and the
+ * {@link Flavor} derived from the field's similarity function: independent of the other vectors in
+ * the segment. Identical vectors thus quantize identically and the quantized data de-duplicates
+ * along with the raw data, shared across all documents and fields of a group with the same flavor.
+ *
+ * <p>For the same reason, a record already computed by a source segment does not change on merge:
+ * when a distinct vector originates from a segment in this format (with the same encoding and
+ * flavor), its quantized record is <b>copied</b> instead of re-reading the raw vector and
+ * re-quantizing it.
+ *
+ * <p>Records follow the {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorsFormat} conventions exactly
+ * (per flavor), so they are scored by the stock {@link
+ * org.apache.lucene.codecs.lucene104.Lucene104ScalarQuantizedVectorScorer}.
+ *
+ * @lucene.experimental
+ */
+record DedupQuantizer(ScalarEncoding encoding) {
+
+  /**
+   * How a vector is prepared and quantized, derived from the field's {@link
+   * VectorSimilarityFunction}. Fields of a group whose similarity functions map to the same flavor
+   * share one quantized record per distinct vector; a group stores one block per flavor in use.
+   */
+  enum Flavor {
+
+    /** Vectors quantized as-is; the additional correction holds the squared norm. */
+    EUCLIDEAN(VectorSimilarityFunction.EUCLIDEAN, false),
+
+    /**
+     * Vectors quantized as-is; the additional correction holds the dot product with the (zero)
+     * centroid, i.e. zero.
+     */
+    DOT_PRODUCT(VectorSimilarityFunction.DOT_PRODUCT, false),
+
+    /**
+     * Vectors are l2-normalized before quantization (for cosine similarity, scored as a dot
+     * product); the additional correction is zero as for {@link #DOT_PRODUCT}.
+     */
+    NORMALIZED(VectorSimilarityFunction.DOT_PRODUCT, true);
+
+    private final VectorSimilarityFunction quantizerFunction;
+    private final boolean normalized;
+
+    Flavor(VectorSimilarityFunction quantizerFunction, boolean normalized) {
+      this.quantizerFunction = quantizerFunction;
+      this.normalized = normalized;
+    }
+
+    static Flavor of(VectorSimilarityFunction function) {
+      return switch (function) {
+        case EUCLIDEAN -> EUCLIDEAN;
+        case DOT_PRODUCT, MAXIMUM_INNER_PRODUCT -> DOT_PRODUCT;
+        case COSINE -> NORMALIZED;
+      };
+    }
+
+    /**
+     * The quantizer producing this flavor's records against a zero centroid. NOTE: {@link
+     * #NORMALIZED} quantizes with {@link VectorSimilarityFunction#DOT_PRODUCT} (identical bytes and
+     * corrections, without the unit-centroid requirement of a cosine-configured quantizer).
+     */
+    OptimizedScalarQuantizer quantizer() {
+      return new OptimizedScalarQuantizer(quantizerFunction);
+    }
+
+    boolean normalized() {
+      return normalized;
+    }
+  }
+
+  /** Supplies the distinct (raw) float vector at a group ordinal. */
+  interface FloatVectorSupplier {
+    float[] get(int ord) throws IOException;
+  }
+
+  /**
+   * The already-quantized record of a distinct vector, held by a source segment at {@code ord} with
+   * the given flavor.
+   */
+  record PreQuantized(QuantizedByteVectorValues values, Flavor flavor, int ord) {}
+
+  /**
+   * Supplies the {@link PreQuantized} record of the distinct vector at a group ordinal, or {@code
+   * null} when its source segment does not hold one.
+   */
+  interface PreQuantizedSupplier {
+    PreQuantized get(int ord) throws IOException;
+  }
+
+  /** The encoding, offset and size of one flavor's quantized data block. */
+  record QuantizedBlock(
+      ScalarEncoding encoding, long quantizedDataOffset, long quantizedDataSize) {}
+
+  /** Writes an empty flavor-block list, for groups without quantized data. */
+  static void writeEmptyGroup(IndexOutput meta) throws IOException {
+    meta.writeVInt(0);
+  }
+
+  /** Reads the flavor-block list of a group, the counterpart of {@link #writeGroup}. */
+  static Map<Flavor, QuantizedBlock> readGroup(IndexInput meta) throws IOException {
+    int numFlavors = meta.readVInt();
+    Map<Flavor, QuantizedBlock> blocks = new EnumMap<>(Flavor.class);
+    for (int i = 0; i < numFlavors; i++) {
+      int flavorOrd = meta.readVInt();
+      if (flavorOrd < 0 || flavorOrd >= Flavor.values().length) {
+        throw new CorruptIndexException("Invalid flavor ordinal: " + flavorOrd, meta);
+      }
+      Flavor flavor = Flavor.values()[flavorOrd];
+      int wireNumber = meta.readVInt();
+      ScalarEncoding encoding =
+          ScalarEncoding.fromWireNumber(wireNumber)
+              .orElseThrow(
+                  () ->
+                      new CorruptIndexException(
+                          "Invalid scalar encoding wire number: " + wireNumber, meta));
+      long offset = meta.readLong();
+      long size = meta.readLong();
+      if (blocks.put(flavor, new QuantizedBlock(encoding, offset, size)) != null) {
+        throw new CorruptIndexException("Duplicate flavor: " + flavor, meta);
+      }
+    }
+    return blocks;
+  }
+
+  /**
+   * Quantizes and writes a group's distinct vectors: one block per flavor, one record per group
+   * ordinal within each block. Wherever a {@code preQuantized} record with a matching encoding and
+   * flavor is available (i.e. the vector originates from a segment in this format), it is copied
+   * as-is; otherwise the raw vector is quantized. The block locations are written to {@code meta};
+   * non-FLOAT32 groups (stored raw only) record an empty list.
+   */
+  void writeGroup(
+      IndexOutput meta,
+      IndexOutput quantizedVectorData,
+      VectorEncoding groupEncoding,
+      int dimension,
+      int numVectors,
+      Set<Flavor> flavors,
+      FloatVectorSupplier vectors,
+      PreQuantizedSupplier preQuantized)
+      throws IOException {
+
+    if (groupEncoding != VectorEncoding.FLOAT32) {
+      writeEmptyGroup(meta);
+      return;
+    }
+
+    meta.writeVInt(flavors.size());
+    for (Flavor flavor : Flavor.values()) {
+      if (flavors.contains(flavor) == false) {
+        continue;
+      }
+      QuantizedBlock block =
+          writeFlavorBlock(
+              quantizedVectorData, flavor, dimension, numVectors, vectors, preQuantized);
+      meta.writeVInt(flavor.ordinal());
+      meta.writeVInt(block.encoding().getWireNumber());
+      meta.writeLong(block.quantizedDataOffset());
+      meta.writeLong(block.quantizedDataSize());
+    }
+  }
+
+  private QuantizedBlock writeFlavorBlock(
+      IndexOutput quantizedVectorData,
+      Flavor flavor,
+      int dimension,
+      int numVectors,
+      FloatVectorSupplier vectors,
+      PreQuantizedSupplier preQuantized)
+      throws IOException {
+
+    OptimizedScalarQuantizer quantizer = flavor.quantizer();
+    float[] zeroCentroid = new float[dimension];
+    float[] normalized = flavor.normalized() ? new float[dimension] : null;
+    byte[] scratch = new byte[encoding.getDiscreteDimensions(dimension)];
+    byte[] packed =
+        switch (encoding) {
+          case UNSIGNED_BYTE, SEVEN_BIT -> scratch;
+          case PACKED_NIBBLE, SINGLE_BIT_QUERY_NIBBLE, DIBIT_QUERY_NIBBLE ->
+              new byte[encoding.getDocPackedLength(scratch.length)];
+        };
+
+    long quantizedDataOffset = quantizedVectorData.alignFilePointer(Float.BYTES);
+    for (int ord = 0; ord < numVectors; ord++) {
+      // The quantized record is a pure function of the raw vector and the flavor: a record
+      // already computed by a source segment can be copied instead of re-quantizing.
+      PreQuantized pre = preQuantized == null ? null : preQuantized.get(ord);
+      if (pre != null && pre.flavor() == flavor && pre.values().getScalarEncoding() == encoding) {
+        // NOTE: read the packed bytes before the corrective terms, which are then served from the
+        // record cached by the read of the packed bytes
+        writeRecord(
+            quantizedVectorData,
+            pre.values().vectorValue(pre.ord()),
+            pre.values().getCorrectiveTerms(pre.ord()));
+        continue;
+      }
+
+      float[] vector = vectors.get(ord);
+      if (flavor.normalized()) {
+        // normalize a copy: the source buffer is shared / owned by the group
+        System.arraycopy(vector, 0, normalized, 0, dimension);

Review Comment:
   do we need to make a copy? I think we are free to normalize `vector` in place?



##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/dedup/DedupVectorValues.java:
##########
@@ -469,7 +475,7 @@ public int get(int ord) {
 
     @Override
     public FieldOrdToGroupOrd copy() {
-      return new FieldOrdToGroupOrdArrayList(fieldOrdToGroupOrd);
+      throw new UnsupportedOperationException("not meant for copying");

Review Comment:
    I guess we just never needed this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on PR #16506:
URL: https://github.com/apache/lucene/pull/16506#issuecomment-5441383833

   Thanks for the review @msokolov!
   
   I'll merge this in a couple of days unless there are further comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Add a scalar-quantized version of the de-duplicating vector format [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on code in PR #16506:
URL: https://github.com/apache/lucene/pull/16506#discussion_r3873211019


##########
lucene/sandbox/src/test/org/apache/lucene/sandbox/codecs/dedup/TestDedupScalarQuantizedVectorsFormat.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.sandbox.codecs.dedup;
+
+import static org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsFormat.DEFAULT_BEAM_WIDTH;
+import static org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsFormat.DEFAULT_MAX_CONN;
+import static org.apache.lucene.index.VectorEncoding.BYTE;
+import static org.apache.lucene.index.VectorEncoding.FLOAT32;
+import static org.apache.lucene.index.VectorSimilarityFunction.COSINE;
+import static org.apache.lucene.index.VectorSimilarityFunction.DOT_PRODUCT;
+import static org.apache.lucene.index.VectorSimilarityFunction.EUCLIDEAN;
+import static org.apache.lucene.index.VectorSimilarityFunction.MAXIMUM_INNER_PRODUCT;
+import static org.hamcrest.Matchers.instanceOf;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.lucene.codecs.KnnVectorsReader;
+import org.apache.lucene.codecs.hnsw.FlatVectorsReader;
+import org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsReader;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.KnnByteVectorField;
+import org.apache.lucene.document.KnnFloatVectorField;
+import org.apache.lucene.document.NumericDocValuesField;
+import org.apache.lucene.index.ByteVectorValues;
+import org.apache.lucene.index.CodecReader;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.FloatVectorValues;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.KnnVectorValues;
+import org.apache.lucene.index.LeafReader;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.search.AcceptDocs;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.search.TopDocs;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.tests.util.LuceneTestCase;
+import org.apache.lucene.tests.util.TestUtil;
+import org.apache.lucene.util.VectorUtil;
+import org.apache.lucene.util.quantization.QuantizedByteVectorValues.ScalarEncoding;
+
+/**
+ * Tests that {@link DedupHnswScalarQuantizedVectorsFormat} stores each distinct vector once, in
+ * both raw and quantized form. General de-duplication behavior is covered by {@link
+ * TestDedupFlatVectorsFormat}; this test focuses on the quantized side.
+ */
+public class TestDedupScalarQuantizedVectorsFormat extends LuceneTestCase {

Review Comment:
   Added now!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org