You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ji...@apache.org on 2026/10/07 11:12:20 UTC
(lucene) branch main updated: Fix the vectors merge mapping from #16749, and map again for Faiss merges (#16769)
This is an automated email from the ASF dual-hosted git repository.
jimczi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/lucene.git
The following commit(s) were added to refs/heads/main by this push:
new dba8f5e9534 Fix the vectors merge mapping from #16749, and map again for Faiss merges (#16769)
dba8f5e9534 is described below
commit dba8f5e9534eedeccb82ef1e2a485bb91b53c8cc
Author: Jim Ferenczi <ji...@apache.org>
AuthorDate: Wed Oct 7 13:12:14 2026 +0200
Fix the vectors merge mapping from #16749, and map again for Faiss merges (#16769)
#16749 made `Lucene99FlatVectorsReader.getMergeInstance()` map the raw vectors again for a merge, instead of changing the advice on the mapping searches read at random. This fixes a few things in that change and covers a reader it missed.
**Map again only when searches read at random.** A merge maps the file again only when searches read it at random and the reader was not opened by a merge itself. Otherwise the search mapping carries no advice a merge reading front to back would suffer from, so a flat format used without a graph keeps reading through the mapping it has.
**Keep the caller's hints.** The merge mapping was built from scratch. It now keeps what the caller said about the file and only replaces the access hint with `SEQUENTIAL`. Reuse stays the format's call: `Lucene104ScalarQuantizedVectorsFormat` marks its raw vectors `NoReuse` and the merge mapping keeps it; `Lucene99HnswVectorsFormat` doesn't, and neither does the merge mapping.
**Release once per merge instance, outside the lock.** Finishing the original reader, or finishing a merge instance twice, released another instance's hold on the mapping. Each merge instance now gives it back once, and the last one closes it outside the lock.
**Faiss.** `FaissKnnVectorsReader` returned `this` from `getMergeInstance()`, so the raw reader's merge instance was never asked for. It now hands merges a reader over the raw reader's merge instance and passes `finishMerge()` on. The Faiss index file is not read by a merge.
`TestVectorsMergeReadAdvice` covers the merge mapping for a graph over raw vectors and for the scalar quantized format, the two cases that must not map again, the caller's hints surviving, sharing between concurrent merge instances, release-once, and close while a merge is open.
---
lucene/CHANGES.txt | 12 +-
.../codecs/lucene99/Lucene99FlatVectorsReader.java | 103 ++++++---
.../lucene99/TestVectorsMergeReadAdvice.java | 243 ++++++++++++++++++++-
.../codecs/faiss/FaissKnnVectorsReader.java | 28 +++
4 files changed, 342 insertions(+), 44 deletions(-)
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index c43a10541f9..ae9c4b42cee 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -495,10 +495,10 @@ Optimizations
ordinal to group ordinal mapping at the minimum bit width required for the largest group ordinal,
instead of a fixed 32-bit width. (Pranshu Shukla)
-* GITHUB#16749: Merges now open the raw vectors for themselves and state that they read the data
- file sequentially, instead of changing the advice on the mapping searches read at random. The
- vectors writers now state what file they are writing, and the raw vectors a quantized format keeps
- only to rescore say so as they are written. (Jim Ferenczi)
+* GITHUB#16749: Merges now open the raw vectors for themselves when searches read them at random,
+ and state that they read the data file sequentially, instead of changing the advice on the mapping
+ searches use. The vectors writers now state what file they are writing, and the raw vectors a
+ quantized format keeps only to rescore say so as they are written. (Jim Ferenczi)
* GITHUB#16622: Merges of quantized vector fields now also open the raw vectors for themselves.
Lucene104ScalarQuantizedVectorsReader, and the backward-codecs Lucene99 and Lucene102 quantized
@@ -610,6 +610,10 @@ Optimizations
* GITHUB#16667: Rewrite BooleanQuery to MatchNoDocsQuery when it has MUST_NOT clause matching one of
its MUST boost or constant score query clauses. (Egor Potemkin)
+* GITHUB#16769: Merges now also open their own mapping of the raw vectors behind the Faiss format,
+ and the merge mapping keeps the hints the file was opened with, changing only how it is read.
+ (Jim Ferenczi)
+
Bug Fixes
---------------------
* GITHUB#10153: Allow configurable buffer size in SegmentingTokenizerBase and recognize whitespace as safe boundary in ThaiTokenizer to avoid splitting words across buffer boundaries. (Kamthorn Krairaksa)
diff --git a/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99FlatVectorsReader.java b/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99FlatVectorsReader.java
index 2754dbac007..91dcb47a6e1 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99FlatVectorsReader.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99FlatVectorsReader.java
@@ -24,6 +24,7 @@ import java.io.FileNotFoundException;
import java.io.IOException;
import java.nio.file.NoSuchFileException;
import java.util.Map;
+import java.util.stream.Stream;
import org.apache.lucene.codecs.CodecUtil;
import org.apache.lucene.codecs.hnsw.FlatVectorsReader;
import org.apache.lucene.codecs.hnsw.FlatVectorsScorer;
@@ -50,7 +51,6 @@ import org.apache.lucene.store.FileDataHint;
import org.apache.lucene.store.FileTypeHint;
import org.apache.lucene.store.IOContext;
import org.apache.lucene.store.IndexInput;
-import org.apache.lucene.store.NoReuseHint;
import org.apache.lucene.util.IOUtils;
import org.apache.lucene.util.RamUsageEstimator;
import org.apache.lucene.util.hnsw.RandomVectorScorer;
@@ -77,6 +77,8 @@ public final class Lucene99FlatVectorsReader extends FlatVectorsReader {
private IndexInput mergeVectorData;
// merge instances handed out and not yet finished
private int mergeInstances;
+ // on a merge instance: whether it gave the mapping back, guarded by the original's lock
+ private boolean finished;
public Lucene99FlatVectorsReader(SegmentReadState state, FlatVectorsScorer scorer)
throws IOException {
@@ -210,6 +212,9 @@ public final class Lucene99FlatVectorsReader extends FlatVectorsReader {
@Override
public FlatVectorsReader getMergeInstance() throws IOException {
+ if (mergeNeedsItsOwnMapping() == false) {
+ return this;
+ }
IndexInput data = original.mergeVectorData();
boolean success = false;
try {
@@ -218,42 +223,50 @@ public final class Lucene99FlatVectorsReader extends FlatVectorsReader {
return mergeInstance;
} finally {
if (success == false) {
- original.releaseMergeVectorData();
+ IOUtils.closeWhileHandlingException(original.release());
}
}
}
/**
- * The vectors as a merge reads them, front to back and once. Advice belongs to a mapping, so a
- * merge maps the file again. Mapped on the first merge, released by {@link #finishMerge()}.
+ * Whether a merge has to map the file again: only when searches read it at random. Otherwise the
+ * mapping searches use carries no advice a merge reading front to back would suffer from, and a
+ * reader a merge opened already reads the file the way a merge does.
+ */
+ private boolean mergeNeedsItsOwnMapping() {
+ return dataContext.context() != IOContext.Context.MERGE
+ && dataContext.hints().contains(DataAccessHint.RANDOM);
+ }
+
+ /**
+ * The vectors as a merge reads them, front to back. Advice belongs to a mapping, so a merge maps
+ * the file again. Mapped on the first merge, released by {@link #finishMerge()}.
*/
private synchronized IndexInput mergeVectorData() throws IOException {
assert original == this;
if (mergeVectorData == null) {
- if (dataContext.context() == IOContext.Context.MERGE) {
- // opened by a merge to begin with, so it already reads the file front to back
+ try {
+ mergeVectorData = directory.openInput(vectorDataFN, mergeContext());
+ } catch (FileNotFoundException | NoSuchFileException _) {
+ // an open reader outlives its files, so fall back to the mapping it already holds
mergeVectorData = vectorData;
- } else {
- try {
- mergeVectorData =
- directory.openInput(
- vectorDataFN,
- IOContext.merge()
- .withHints(
- FileTypeHint.DATA,
- FileDataHint.KNN_VECTORS,
- DataAccessHint.SEQUENTIAL,
- NoReuseHint.INSTANCE));
- } catch (FileNotFoundException | NoSuchFileException _) {
- // an open reader outlives its files, so fall back to the mapping it already holds
- mergeVectorData = vectorData;
- }
}
}
mergeInstances++;
return mergeVectorData;
}
+ /** The caller's context as a merge reading the file front to back: only the access changes. */
+ private IOContext mergeContext() {
+ return IOContext.merge()
+ .withHints(
+ Stream.concat(
+ dataContext.hints().stream()
+ .filter(hint -> hint instanceof DataAccessHint == false),
+ Stream.of(DataAccessHint.SEQUENTIAL))
+ .toArray(IOContext.FileOpenHint[]::new));
+ }
+
private FieldEntry getFieldEntryOrThrow(String field) {
final FieldInfo info = fieldInfos.fieldInfo(field);
final FieldEntry entry;
@@ -378,32 +391,56 @@ public final class Lucene99FlatVectorsReader extends FlatVectorsReader {
/**
* Closes the mapping a merge used, once no merge instance holds it. A later merge maps the file
- * again.
+ * again. Only a merge instance holds the mapping, and it gives it back once: finishing the reader
+ * it came from, or finishing it again, releases nothing.
*/
@Override
public void finishMerge() throws IOException {
- original.releaseMergeVectorData();
+ if (original != this) {
+ IOUtils.close(original.releaseMergeVectorData(this));
+ }
}
- private synchronized void releaseMergeVectorData() throws IOException {
- assert original == this;
- if (--mergeInstances > 0) {
- return;
+ /** Gives back the hold of {@code mergeInstance}, once; returns the mapping to close, if any. */
+ private synchronized IndexInput releaseMergeVectorData(Lucene99FlatVectorsReader mergeInstance) {
+ assert original == this && mergeInstance.original == this;
+ if (mergeInstance.finished) {
+ return null;
}
- if (mergeVectorData != null && mergeVectorData != vectorData) {
- mergeVectorData.close();
+ mergeInstance.finished = true;
+ return release();
+ }
+
+ /**
+ * Gives back one hold on the mapping. Once none is left, returns it for the caller to close
+ * outside the lock.
+ */
+ private synchronized IndexInput release() {
+ assert original == this && mergeInstances > 0;
+ if (--mergeInstances > 0) {
+ return null;
}
+ IndexInput toClose = mergeVectorData == vectorData ? null : mergeVectorData;
mergeVectorData = null;
+ return toClose;
}
@Override
public void close() throws IOException {
- IOUtils.close(vectorData, mergeVectorDataToClose());
+ IOUtils.close(vectorData, takeMergeVectorData());
}
- /** The mapping a merge opened, read under the lock that guards it, closed outside it. */
- private synchronized IndexInput mergeVectorDataToClose() {
- return original == this && mergeVectorData != vectorData ? mergeVectorData : null;
+ /**
+ * The mapping a merge opened, taken under the lock that guards it so that a merge finishing later
+ * does not close it again, and closed outside it.
+ */
+ private synchronized IndexInput takeMergeVectorData() {
+ if (original != this || mergeVectorData == vectorData) {
+ return null;
+ }
+ IndexInput toClose = mergeVectorData;
+ mergeVectorData = null;
+ return toClose;
}
private record FieldEntry(
diff --git a/lucene/core/src/test/org/apache/lucene/codecs/lucene99/TestVectorsMergeReadAdvice.java b/lucene/core/src/test/org/apache/lucene/codecs/lucene99/TestVectorsMergeReadAdvice.java
index fa0317de6d0..d501ac47023 100644
--- a/lucene/core/src/test/org/apache/lucene/codecs/lucene99/TestVectorsMergeReadAdvice.java
+++ b/lucene/core/src/test/org/apache/lucene/codecs/lucene99/TestVectorsMergeReadAdvice.java
@@ -23,16 +23,23 @@ import java.util.Set;
import java.util.concurrent.atomic.AtomicBoolean;
import org.apache.lucene.codecs.KnnVectorsFormat;
import org.apache.lucene.codecs.KnnVectorsReader;
+import org.apache.lucene.codecs.hnsw.FlatVectorScorerUtil;
+import org.apache.lucene.codecs.hnsw.FlatVectorsReader;
import org.apache.lucene.codecs.lucene104.Lucene104HnswScalarQuantizedVectorsFormat;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.KnnFloatVectorField;
import org.apache.lucene.index.CodecReader;
import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.FloatVectorValues;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.SegmentReadState;
+import org.apache.lucene.index.SegmentReader;
import org.apache.lucene.index.VectorSimilarityFunction;
import org.apache.lucene.store.DataAccessHint;
import org.apache.lucene.store.Directory;
+import org.apache.lucene.store.FileDataHint;
+import org.apache.lucene.store.FileTypeHint;
import org.apache.lucene.store.FilterDirectory;
import org.apache.lucene.store.FilterIndexInput;
import org.apache.lucene.store.IOContext;
@@ -51,14 +58,18 @@ public class TestVectorsMergeReadAdvice extends LuceneTestCase {
private static final int DIM = 8;
public void testMergeOpensItsOwnVectors() throws Exception {
- assertMergeOpensItsOwnVectors(new Lucene99HnswVectorsFormat());
+ assertMergeOpensItsOwnVectors(new Lucene99HnswVectorsFormat(), false);
}
public void testQuantizedMergeOpensItsOwnVectors() throws Exception {
- assertMergeOpensItsOwnVectors(new Lucene104HnswScalarQuantizedVectorsFormat());
+ assertMergeOpensItsOwnVectors(new Lucene104HnswScalarQuantizedVectorsFormat(), true);
}
- private static void assertMergeOpensItsOwnVectors(KnnVectorsFormat format) throws Exception {
+ /**
+ * @param noReuse whether the format opens its raw vectors with {@link NoReuseHint}
+ */
+ private static void assertMergeOpensItsOwnVectors(KnnVectorsFormat format, boolean noReuse)
+ throws Exception {
Opens opens = new Opens();
try (Directory dir = new RecordingDirectory(newDirectory(), opens)) {
IndexWriterConfig iwc = new IndexWriterConfig();
@@ -88,6 +99,10 @@ public class TestVectorsMergeReadAdvice extends LuceneTestCase {
for (Open open : sequential) {
assertTrue("the merge did not read the vectors it opened: " + open, open.read());
assertTrue("the merge kept its vectors open: " + open, open.closed());
+ assertEquals(
+ "the merge mapping keeps what the format said: " + open,
+ noReuse,
+ open.hints().contains(NoReuseHint.INSTANCE));
}
}
}
@@ -99,6 +114,129 @@ public class TestVectorsMergeReadAdvice extends LuceneTestCase {
}
}
+ /** A reader a merge opened already reads the vectors the way a merge does. */
+ public void testAMergeReadsThroughTheReaderItOpened() throws Exception {
+ Opens opens = new Opens();
+ try (Directory dir = new RecordingDirectory(newDirectory(), opens)) {
+ IndexWriterConfig iwc = new IndexWriterConfig();
+ iwc.setCodec(TestUtil.alwaysKnnVectorsFormat(new Lucene99HnswVectorsFormat()));
+ iwc.setUseCompoundFile(false);
+ try (IndexWriter w = new IndexWriter(dir, iwc)) {
+ for (int segment = 0; segment < 2; segment++) {
+ for (int i = 0; i < 16; i++) {
+ Document doc = new Document();
+ doc.add(
+ new KnnFloatVectorField("field", vector(), VectorSimilarityFunction.DOT_PRODUCT));
+ w.addDocument(doc);
+ }
+ w.commit();
+ }
+ // no reader is open, so the merge opens the segments itself, with a merge context
+ opens.clear();
+ w.forceMerge(1);
+ }
+ assertEquals(
+ "the merge mapped vectors its own readers already map: " + opens,
+ List.of(),
+ sequentialOpens(opens));
+ assertEquals("the merge re-advised the vectors: " + opens, List.of(), opens.advised());
+ }
+ }
+
+ /** Searches that do not read the vectors at random leave nothing for a merge to undo. */
+ public void testNoMappingWhenSearchesDoNotReadAtRandom() throws Exception {
+ Opens opens = new Opens();
+ try (Directory dir = new RecordingDirectory(newDirectory(), opens)) {
+ IndexWriterConfig iwc = new IndexWriterConfig();
+ iwc.setCodec(TestUtil.alwaysKnnVectorsFormat(new Lucene99HnswVectorsFormat()));
+ iwc.setUseCompoundFile(false);
+ try (IndexWriter w = new IndexWriter(dir, iwc)) {
+ for (int i = 0; i < 16; i++) {
+ Document doc = new Document();
+ doc.add(new KnnFloatVectorField("field", vector(), VectorSimilarityFunction.DOT_PRODUCT));
+ w.addDocument(doc);
+ }
+ w.commit();
+ }
+ try (DirectoryReader reader = DirectoryReader.open(dir)) {
+ SegmentReader segment = (SegmentReader) getOnlyLeafReader(reader);
+ try (FlatVectorsReader flat = flatReader(dir, segment, IOContext.DEFAULT)) {
+ opens.clear();
+ assertSame(flat, flat.getMergeInstance());
+ flat.finishMerge();
+ assertEquals("nothing to map again: " + opens, List.of(), opens.all());
+ }
+ }
+ }
+ }
+
+ /** The merge mapping changes how the file is read and keeps everything else the caller said. */
+ public void testTheMergeMappingKeepsWhatTheCallerSaid() throws Exception {
+ Opens opens = new Opens();
+ try (Directory dir = new RecordingDirectory(newDirectory(), opens)) {
+ IndexWriterConfig iwc = new IndexWriterConfig();
+ iwc.setCodec(TestUtil.alwaysKnnVectorsFormat(new Lucene99HnswVectorsFormat()));
+ iwc.setUseCompoundFile(false);
+ try (IndexWriter w = new IndexWriter(dir, iwc)) {
+ for (int i = 0; i < 16; i++) {
+ Document doc = new Document();
+ doc.add(new KnnFloatVectorField("field", vector(), VectorSimilarityFunction.DOT_PRODUCT));
+ w.addDocument(doc);
+ }
+ w.commit();
+ }
+ try (DirectoryReader reader = DirectoryReader.open(dir)) {
+ SegmentReader segment = (SegmentReader) getOnlyLeafReader(reader);
+ IOContext searchContext =
+ IOContext.DEFAULT.withHints(
+ DataAccessHint.RANDOM, NoReuseHint.INSTANCE, CallerHint.INSTANCE);
+ try (FlatVectorsReader flat = flatReader(dir, segment, searchContext)) {
+ opens.clear();
+ FlatVectorsReader mergeInstance = flat.getMergeInstance();
+ List<Open> mapped = sequentialOpens(opens);
+ assertEquals("one mapping for the merge: " + opens, 1, mapped.size());
+ Set<IOContext.FileOpenHint> hints = mapped.get(0).hints();
+ assertTrue("the caller's hint is kept: " + hints, hints.contains(CallerHint.INSTANCE));
+ assertTrue(hints.contains(NoReuseHint.INSTANCE));
+ assertTrue(hints.contains(FileTypeHint.DATA));
+ assertTrue(hints.contains(FileDataHint.KNN_VECTORS));
+ assertFalse(
+ "searches read at random, not the merge: " + hints,
+ hints.contains(DataAccessHint.RANDOM));
+ mergeInstance.finishMerge();
+ }
+ }
+ }
+ }
+
+ /** A hint only the caller knows about. */
+ private enum CallerHint implements IOContext.FileOpenHint {
+ INSTANCE
+ }
+
+ /**
+ * A flat vectors reader over the only vectors file of {@code segment}, opened with {@code
+ * context}.
+ */
+ private static FlatVectorsReader flatReader(
+ Directory dir, SegmentReader segment, IOContext context) throws IOException {
+ String vectorData = null;
+ for (String file : segment.getSegmentInfo().files()) {
+ if (file.endsWith("." + Lucene99FlatVectorsFormat.VECTOR_DATA_EXTENSION)) {
+ vectorData = file;
+ }
+ }
+ assertNotNull(vectorData);
+ String segmentName = segment.getSegmentInfo().info.name;
+ // the per-field suffix, between the segment name and the extension
+ String suffix = vectorData.substring(segmentName.length() + 1, vectorData.lastIndexOf('.'));
+ SegmentReadState state =
+ new SegmentReadState(
+ dir, segment.getSegmentInfo().info, segment.getFieldInfos(), context, suffix);
+ return new Lucene99FlatVectorsFormat(FlatVectorScorerUtil.getLucene99FlatVectorsScorer())
+ .fieldsReader(state);
+ }
+
public void testTheMappingIsSharedThenReleasedByTheLastMergeInstance() throws Exception {
assertMappingIsSharedThenReleasedByTheLastMergeInstance(new Lucene99HnswVectorsFormat());
}
@@ -151,6 +289,100 @@ public class TestVectorsMergeReadAdvice extends LuceneTestCase {
}
}
+ /**
+ * Fields sharing a reader each ask it for a merge instance, so one segment has several merge
+ * instances at once. They share one mapping, each through a clone of its own.
+ */
+ public void testFieldsSharingAReaderShareOneMapping() throws Exception {
+ Opens opens = new Opens();
+ try (Directory dir = new RecordingDirectory(newDirectory(), opens)) {
+ IndexWriterConfig iwc = new IndexWriterConfig();
+ iwc.setCodec(TestUtil.alwaysKnnVectorsFormat(new Lucene99HnswVectorsFormat()));
+ iwc.setUseCompoundFile(false);
+ int segments = 3;
+ try (IndexWriter w = new IndexWriter(dir, iwc)) {
+ for (int segment = 0; segment < segments; segment++) {
+ for (int i = 0; i < 16; i++) {
+ Document doc = new Document();
+ doc.add(new KnnFloatVectorField("a", vector(), VectorSimilarityFunction.DOT_PRODUCT));
+ doc.add(new KnnFloatVectorField("b", vector(), VectorSimilarityFunction.DOT_PRODUCT));
+ doc.add(new KnnFloatVectorField("c", vector(), VectorSimilarityFunction.DOT_PRODUCT));
+ w.addDocument(doc);
+ }
+ w.commit();
+ }
+ try (DirectoryReader reader = DirectoryReader.open(w)) {
+ assertEquals(segments, reader.leaves().size());
+ opens.clear();
+ w.forceMerge(1);
+
+ List<Open> sequential = sequentialOpens(opens);
+ assertEquals("one mapping per merged segment: " + opens, segments, sequential.size());
+ for (Open open : sequential) {
+ assertTrue("released once every field was done: " + open, open.closed());
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * A merge instance gives the mapping back once. Finishing it again, or finishing the reader it
+ * came from, must not release the mapping another merge instance still reads.
+ */
+ public void testOnlyAMergeInstanceReleasesTheMappingAndOnlyOnce() throws Exception {
+ Opens opens = new Opens();
+ try (Directory dir = new RecordingDirectory(newDirectory(), opens)) {
+ IndexWriterConfig iwc = new IndexWriterConfig();
+ iwc.setCodec(TestUtil.alwaysKnnVectorsFormat(new Lucene99HnswVectorsFormat()));
+ iwc.setUseCompoundFile(false);
+ try (IndexWriter w = new IndexWriter(dir, iwc)) {
+ for (int i = 0; i < 16; i++) {
+ Document doc = new Document();
+ doc.add(new KnnFloatVectorField("field", vector(), VectorSimilarityFunction.DOT_PRODUCT));
+ w.addDocument(doc);
+ }
+ w.commit();
+ }
+ List<Open> mapped;
+ KnnVectorsReader third;
+ try (DirectoryReader reader = DirectoryReader.open(dir)) {
+ KnnVectorsReader vectors =
+ ((CodecReader) getOnlyLeafReader(reader))
+ .getVectorReader()
+ .unwrapReaderForField("field");
+ opens.clear();
+
+ KnnVectorsReader first = vectors.getMergeInstance();
+ KnnVectorsReader second = vectors.getMergeInstance();
+ mapped = sequentialOpens(opens);
+ assertEquals("one mapping for both merge instances: " + opens, 1, mapped.size());
+
+ first.finishMerge();
+ first.finishMerge();
+ assertFalse("finishing twice released another's hold: " + opens, mapped.get(0).closed());
+
+ vectors.finishMerge();
+ assertFalse("the reader itself holds nothing to release: " + opens, mapped.get(0).closed());
+
+ // the second instance still reads through the mapping
+ FloatVectorValues values = second.getFloatVectorValues("field");
+ assertEquals(DIM, values.vectorValue(0).length);
+
+ second.finishMerge();
+ assertTrue("released by the last instance: " + opens, mapped.get(0).closed());
+
+ // a merge instance left unfinished when the reader closes
+ third = vectors.getMergeInstance();
+ mapped = sequentialOpens(opens);
+ assertEquals(2, mapped.size());
+ }
+ assertTrue("closed with the reader: " + opens, mapped.get(1).closed());
+ // and finishing it afterwards does not close it a second time
+ third.finishMerge();
+ }
+ }
+
/** The mapping goes with the reader, even if no merge ever says it is finished. */
public void testTheMappingIsClosedWithTheReader() throws Exception {
Opens opens = new Opens();
@@ -241,7 +473,7 @@ public class TestVectorsMergeReadAdvice extends LuceneTestCase {
return v;
}
- /** The opens a merge made for itself: the vector data, read front to back and once. */
+ /** The opens a merge made for itself: the vector data, read front to back. */
private static List<Open> sequentialOpens(Opens opens) {
List<Open> sequential = new ArrayList<>();
for (Open open : opens.all()) {
@@ -249,9 +481,6 @@ public class TestVectorsMergeReadAdvice extends LuceneTestCase {
if (open.name().endsWith(".vec")
&& open.hint() == DataAccessHint.SEQUENTIAL
&& open.hints().contains(ReadOnceHint.INSTANCE) == false) {
- assertTrue(
- "a merge reads the vectors once and does not come back: " + open,
- open.hints().contains(NoReuseHint.INSTANCE));
assertSame(
"the open says a merge is reading, so a directory can route it: " + open,
IOContext.Context.MERGE,
diff --git a/lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/faiss/FaissKnnVectorsReader.java b/lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/faiss/FaissKnnVectorsReader.java
index 49386996fa7..1c83b45e6b1 100644
--- a/lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/faiss/FaissKnnVectorsReader.java
+++ b/lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/faiss/FaissKnnVectorsReader.java
@@ -59,6 +59,16 @@ final class FaissKnnVectorsReader extends KnnVectorsReader {
private final Map<String, FaissLibrary.Index> indexMap;
private boolean closed;
+ /**
+ * Reads the same indexes as {@code reader}, with the raw vectors read by {@code
+ * rawVectorsReader}.
+ */
+ private FaissKnnVectorsReader(FaissKnnVectorsReader reader, FlatVectorsReader rawVectorsReader) {
+ this.rawVectorsReader = rawVectorsReader;
+ this.data = reader.data;
+ this.indexMap = reader.indexMap;
+ }
+
public FaissKnnVectorsReader(SegmentReadState state, FlatVectorsReader rawVectorsReader)
throws IOException {
this.rawVectorsReader = rawVectorsReader;
@@ -208,6 +218,24 @@ final class FaissKnnVectorsReader extends KnnVectorsReader {
return rawVectorsReader.getVectorCount(fieldInfo);
}
+ /**
+ * A merge rebuilds the index from the raw vectors, so it reads them the way the raw reader's
+ * merge instance does. The index itself is not read by a merge.
+ */
+ @Override
+ public KnnVectorsReader getMergeInstance() throws IOException {
+ FlatVectorsReader rawMergeInstance = rawVectorsReader.getMergeInstance();
+ if (rawMergeInstance == rawVectorsReader) {
+ return this;
+ }
+ return new FaissKnnVectorsReader(this, rawMergeInstance);
+ }
+
+ @Override
+ public void finishMerge() throws IOException {
+ rawVectorsReader.finishMerge();
+ }
+
@Override
public void close() throws IOException {
if (closed == false) {