You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "jeho-rpls (via GitHub)" <gi...@apache.org> on 2026/07/07 03:39:24 UTC
[I] HNSW graph construction during merge never checks for abort, blocking IndexWriter#rollback for tens of minutes [lucene]
jeho-rpls opened a new issue, #16367:
URL: https://github.com/apache/lucene/issues/16367
### Description
### Summary
`IndexWriter#abortMerges()` sets the abort flag on running merges and waits for them to finish, relying on merges observing the flag periodically (the comment in `abortMerges` says *"It should not take very long because they periodically check if they are aborted."*). That assumption holds for I/O-bound merge phases — every write goes through the merge rate limiter, which checks for abort — and since #16264 / #16281, `checkIntegrity()` is also covered.
HNSW graph construction breaks this assumption: `HnswGraphBuilder#addVectors` is a pure-CPU loop that performs no writes and never checks `OneMerge#isAborted()`. For large vector segments this phase runs for **10–25 minutes single-threaded**, during which `rollback()` / `abortMerges()` block unconditionally.
We believe this is now the last significant blind spot in merge abort handling, and it is the dominant one for vector-heavy indexes: in our measurements graph construction is **93.6% of merge CPU**, while the checksum phase addressed by #16281 is 0.1%.
### Production impact
Elasticsearch 9.4.2 (Lucene 10.4.0) cluster, HNSW-indexed dense vectors (1M × 384d float32 + 575k × 96d quantized per shard). During a routine rolling restart, each data node's shutdown blocked on `abortMerges` waiting for an in-flight HNSW merge, for **24–31 minutes per node** — exceeding the 30-minute termination grace period, so pods had to be force-killed.
Aborting thread:
```
"elasticsearch[...][generic][T#…]" TIMED_WAITING
at java.lang.Object.wait0(java.base/Native Method)
at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:5571)
at org.apache.lucene.index.IndexWriter.abortMerges(IndexWriter.java:2760)
at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2514)
at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2483)
```
The single merge thread it waits for (RUNNABLE the whole time, no writes):
```
"elasticsearch[...][merge][T#3]" RUNNABLE
... (vector scoring frames elided)
at org.apache.lucene.util.hnsw.HnswGraphSearcher.searchLevel(HnswGraphSearcher.java:342)
at org.apache.lucene.util.hnsw.HnswGraphBuilder.addGraphNodeInternal(HnswGraphBuilder.java:312)
at org.apache.lucene.util.hnsw.HnswGraphBuilder.addGraphNode(HnswGraphBuilder.java:354)
at org.apache.lucene.util.hnsw.HnswGraphBuilder.addVectors(HnswGraphBuilder.java:233)
at org.apache.lucene.util.hnsw.HnswGraphBuilder.build(HnswGraphBuilder.java:201)
at org.apache.lucene.util.hnsw.IncrementalHnswGraphMerger.merge(IncrementalHnswGraphMerger.java:218)
at org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.mergeOneField(Lucene99HnswVectorsWriter.java:449)
at org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.mergeOneField(PerFieldKnnVectorsFormat.java:126)
at org.apache.lucene.codecs.KnnVectorsWriter.merge(KnnVectorsWriter.java:105)
at org.apache.lucene.index.SegmentMerger.mergeVectorValues(SegmentMerger.java:271)
...
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5323)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4784)
```
### Measurements
Instrumenting one incident-class merge (TieredMergePolicy deletes-purging rewrite of a ~3 GB segment, 1.9M docs, JFR + async-profiler):
| Phase | Wall time | Abort-checked today? |
|---|---|---|
| checkIntegrity (checksums) | seconds (0.1% CPU) | yes on main (#16264 / #16281) |
| stored fields / postings / doc values / vector data copy | ~8 min | yes (rate limiter on writes) |
| **HNSW graph construction (both fields)** | **~14.5 min** | **no** |
| HNSW graph serialization + segment finish | ~1 min | yes (writes) |
| total | 23.5 min | |
Graph construction = 93.6% of merge CPU samples (higher than its wall-time share because the copy phases are I/O-bound). `main` has the same shape: `HnswGraphBuilder#addVectors` contains no abort/cancellation hook.
A standalone reproduction below (lucene-core only, 160k × 384d vectors): `rollback()` called 8s into a `forceMerge` blocks for **68.6s** on stock 10.4.0 — the entire remaining graph build.
<details>
<summary>Reproduction (lucene-core 10.4.0 on the classpath, ~1 min indexing + ~1 min to observe the block)</summary>
```java
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Random;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.KnnFloatVectorField;
import org.apache.lucene.index.ConcurrentMergeScheduler;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.NoMergePolicy;
import org.apache.lucene.store.FSDirectory;
/** rollback() while an HNSW merge is building its graph blocks until the build finishes. */
public class MiniRepro {
public static void main(String[] args) throws Exception {
int dim = 384, segs = 16, perSeg = 10000;
long sleepMs = args.length > 0 ? Long.parseLong(args[0]) : 8000;
Path dir = Files.createTempDirectory("hnsw-abort-repro");
try (FSDirectory d = FSDirectory.open(dir)) {
IndexWriterConfig cfg = new IndexWriterConfig();
cfg.setUseCompoundFile(false);
cfg.setMergePolicy(NoMergePolicy.INSTANCE);
try (IndexWriter w = new IndexWriter(d, cfg)) {
Random r = new Random(42);
for (int s = 0; s < segs; s++) {
for (int i = 0; i < perSeg; i++) {
Document doc = new Document();
float[] v = new float[dim];
for (int j = 0; j < dim; j++) v[j] = r.nextFloat();
doc.add(new KnnFloatVectorField("v", v));
w.addDocument(doc);
}
w.flush();
}
w.commit();
}
IndexWriterConfig cfg2 = new IndexWriterConfig();
cfg2.setMergeScheduler(new ConcurrentMergeScheduler());
IndexWriter w2 = new IndexWriter(d, cfg2);
long m0 = System.nanoTime();
Thread bg = new Thread(() -> {
try {
w2.forceMerge(1);
System.out.printf("[bg] forceMerge completed normally in %.1fs%n",
(System.nanoTime() - m0) / 1e9);
} catch (Throwable t) {
System.out.printf("[bg] forceMerge exception after %.1fs: %s%n",
(System.nanoTime() - m0) / 1e9, t);
}
});
bg.start();
Thread.sleep(sleepMs);
System.out.printf("calling rollback() at t=%.1fs after merge start%n",
(System.nanoTime() - m0) / 1e9);
long r0 = System.nanoTime();
w2.rollback();
System.out.printf("### rollback() returned in %.2fs ###%n", (System.nanoTime() - r0) / 1e9);
bg.join();
}
}
}
```
Observed on 10.4.0 (Apple M-series, JDK 23, `--add-modules jdk.incubator.vector`):
```
calling rollback() at t=8.0s after merge start
### rollback() returned in 68.63s ###
```
With a periodic abort check added to `HnswGraphBuilder#addVectors` (same machine, same seed):
```
calling rollback() at t=8.0s after merge start
[bg] forceMerge exception after 8.2s: ... MergeAbortedException ... [ABORTED]
### rollback() returned in 0.20s ###
```
Note: on 10.4.0 this reproduction always takes the full-rebuild path (`HnswGraphBuilder#addVectors`), because `PerFieldKnnVectorsFormat.FieldsReader` does not implement `HnswGraphProvider` there, so graph reuse never kicks in under the default codec. On `main`, where the wrapper does implement it, clean segments take the `MergingHnswGraphBuilder` join path instead — which also has no abort checks (see step 3 below), so the repro demonstrates the block either way.
</details>
### Prior reports
- #11694 (LUCENE-10658, 2022): general request for abortable merges; still open. Elasticsearch's side (elastic/elasticsearch#88055) was closed pointing to it.
- #13354: same pathology, closed after #16264 / #16281 made *checksumming* abortable — which is 0.1% of this workload.
- #13822: proposed plumbing `OneMergeProgress` into `MergeState` for exactly this scenario (HNSW stack in the report), but received no review and its plumbing is now superseded by `MergeState#oneMerge` (#16264).
### Proposed fix
Extend the pattern established by #16264 / #16281 to the graph build path:
1. The plumbing already exists on main: `MergeState#oneMerge` and the null-safe `MergeState#checkAborted()` (#16264), which `KnnVectorsWriter#merge` already invokes once per reader before merging. What's missing is handing that same check down from `Lucene99HnswVectorsWriter#mergeOneField` through the HNSW graph merger into the builders.
2. In `HnswGraphBuilder#addVectors`, invoke it periodically — e.g. every 1024 nodes; each node insertion costs hundreds of vector comparisons, so the overhead is unmeasurable. (This and step 3 could likely be consolidated into a single check in the shared node-insertion path, `addGraphNode`/`addGraphNodeInternal` — happy to follow whichever reviewers prefer.)
3. Cover the incremental-merge path too (`MergingHnswGraphBuilder#build`'s remainder-insertion loop and `updateGraph`), which inserts via `addGraphNode` directly.
4. Same null contract as `checkIntegrity(OneMerge)`: flush-time graph builds pass null and are unaffected.
We validated the approach with a patched 10.4.0 in the affected cluster (same periodic check in `addVectors`; wired through a ThreadLocal shim only because 10.4 predates `MergeState#oneMerge` — the proposal above uses the explicit plumbing instead): node shutdown during a live graph build went from 24–31 min (grace exceeded, force kill) to **2.2 s** (measured between the node's "stopping" and "stopped" log lines), aborted merges clean up through the normal `MergeAbortedException` path, and undisturbed merges complete unchanged.
Happy to contribute a PR with a test (rollback during HNSW merge must abort promptly) if this direction sounds right.
### Version and environment details
```
Lucene 10.4.0 (bundled in Elasticsearch 9.4.2); addVectors on main also lacks abort checks.
OpenJDK 26.0.1, Linux aarch64, single-threaded merges (no intra-merge executor).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] HNSW graph construction during merge never checks for abort, blocking IndexWriter#rollback for tens of minutes [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov closed issue #16367: HNSW graph construction during merge never checks for abort, blocking IndexWriter#rollback for tens of minutes
URL: https://github.com/apache/lucene/issues/16367
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] HNSW graph construction during merge never checks for abort, blocking IndexWriter#rollback for tens of minutes [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #16367:
URL: https://github.com/apache/lucene/issues/16367#issuecomment-4969965633
Whoa, this is a great catch -- thanks @jeho-rpls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org