You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@asterixdb.apache.org by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu> on 2026/08/14 02:59:50 UTC
[S] Change in asterixdb[master]: [ASTERIXDB-3760][RT] Release LOAD I/O op when vector index build fails
From Ali Alsuliman <al...@gmail.com>:
Ali Alsuliman has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21550?usp=email )
Change subject: [ASTERIXDB-3760][RT] Release LOAD I/O op when vector index build fails
......................................................................
[ASTERIXDB-3760][RT] Release LOAD I/O op when vector index build fails
- user model changes: no
- storage format changes: no
- interface changes: no
A failed vector index build wedged the node instead of returning an
error. CREATE INDEX ... TYPE VTREE never came back, and every later
index operation on that node blocked behind it, with no timeout.
VTreeStaticStructureCreator..Descriptor.buildStaticStructureOnPartition
calls ILSMIndex.createBulkLoader(), which declares an active LOAD I/O
operation on the dataset (AbstractLSMIndex.createBulkLoader ->
ioOpCallback.scheduled -> DatasetInfo.declareActiveIOOperation). That
declaration is released only by the loader's end() or abort(). The
method called end() on the success path and had no abort() anywhere, so
any failure between createBulkLoader() and end() leaked the declaration
permanently, once per storage partition.
A leaked declaration is not a slow leak. The next
DatasetLifecycleManager.unregister() -- in practice the drop that rolls
the failed CREATE INDEX back -- blocks forever in
DatasetInfo.waitForIO() waiting for numActiveIOOps to reach zero, and
it does so while holding the DatasetLifecycleManager monitor, so the
remaining partitions on that node pile up behind the monitor.
Wrap the load in the end()/abort() pattern that
IndexBulkLoadOperatorNodePushable.closeBulkLoaders() already uses.
Note that endInvoked is set BEFORE end() rather than after: end()
releases the declaration in its own finally even when it throws, so
aborting a failed end() would release it twice and drive numActiveIOOps
negative, which waitForIO() reports as an IllegalStateException. The
abort itself is best effort and never masks the failure that triggered
it.
Reproducer (8 storage partitions, six rows, so one partition receives
no rows and its k-means yields no centroids):
CREATE COLLECTION s.movies PRIMARY KEY (id: int);
INSERT INTO s.movies ([ ...six rows... ]);
ANALYZE COLLECTION s.movies WITH {"sample-seed": "1000"};
CREATE INDEX i ON s.movies(embedding VECTOR) TYPE VTREE
WITH { "dimension": 3, "similarity": "euclidean", "num_clusters": 2,
"train_list_fraction": 1.0 } EXCLUDE UNKNOWN KEY;
Before: the statement never returns (killed after 21 minutes); a thread
dump shows two partitions parked in DatasetInfo.waitForIO() holding
their DatasetLifecycleManager monitor and the rest blocked on it.
After: the statement fails in 2.5s with "Cannot build vector index: no
training vectors were found".
Whether an empty partition should fail the build at all is a separate
question and is deliberately left as-is here.
Ext-ref: MB-72975
Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
Change-Id: I176d8077e4555069b5c183143eaab2bf03e019bd
---
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/VTreeStaticStructureCreatorOperatorDescriptor.java
1 file changed, 36 insertions(+), 8 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/50/21550/1
diff --git a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/VTreeStaticStructureCreatorOperatorDescriptor.java b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/VTreeStaticStructureCreatorOperatorDescriptor.java
index f9f1772..91359e7 100644
--- a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/VTreeStaticStructureCreatorOperatorDescriptor.java
+++ b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/VTreeStaticStructureCreatorOperatorDescriptor.java
@@ -71,6 +71,7 @@
import org.apache.hyracks.storage.common.IIndexBulkLoader;
import org.apache.hyracks.storage.common.IResource;
import org.apache.hyracks.storage.common.LocalResource;
+import org.apache.hyracks.util.annotations.AiProvenance;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -547,6 +548,7 @@
/**
* Build the static structure on a single storage partition.
*/
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.ASSISTED, notes = "Release the LOAD I/O declaration on the failure path")
private void buildStaticStructureOnPartition(int storagePartition, List<Integer> clustersPerLevel,
List<List<Integer>> centroidsPerCluster, List<ITupleReference> convertedTuples)
throws HyracksDataException {
@@ -582,15 +584,41 @@
IIndexBulkLoader partitionBulkLoader =
partitionLsmIndex.createBulkLoader(fillFactor, false, 0L, false, parameters);
- int totalTuplesProcessed = 0;
- for (ITupleReference convertedTuple : convertedTuples) {
- partitionBulkLoader.add(convertedTuple);
- totalTuplesProcessed++;
- }
+ // createBulkLoader() has already declared an active LOAD I/O operation on the
+ // dataset (AbstractLSMIndex.createBulkLoader -> ioOpCallback.scheduled). That
+ // declaration is released ONLY by end() or abort(). Leaving this scope by any other
+ // route leaks it permanently, and a leaked I/O op is not a slow leak: the next
+ // DatasetLifecycleManager.unregister() -- for instance the drop that rolls this
+ // CREATE INDEX back -- blocks forever in DatasetInfo.waitForIO() while holding the
+ // DatasetLifecycleManager monitor, which wedges every other index operation on the
+ // node with no timeout. Mirror IndexBulkLoadOperatorNodePushable.closeBulkLoaders().
+ //
+ // endInvoked is set BEFORE end() on purpose: end() releases the declaration in its
+ // own finally even when it throws, so aborting a failed end() would release it a
+ // second time and drive numActiveIOOps negative.
+ boolean endInvoked = false;
+ try {
+ int totalTuplesProcessed = 0;
+ for (ITupleReference convertedTuple : convertedTuples) {
+ partitionBulkLoader.add(convertedTuple);
+ totalTuplesProcessed++;
+ }
- partitionBulkLoader.end();
- LOGGER.info("Static structure finalized on storage partition {} ({} tuples)", storagePartition,
- totalTuplesProcessed);
+ endInvoked = true;
+ partitionBulkLoader.end();
+ LOGGER.info("Static structure finalized on storage partition {} ({} tuples)",
+ storagePartition, totalTuplesProcessed);
+ } finally {
+ if (!endInvoked) {
+ try {
+ partitionBulkLoader.abort();
+ } catch (Exception abortFailure) {
+ // Never mask the failure that brought us here; the abort is best effort.
+ LOGGER.warn("Failed to abort the static-structure bulk load on storage "
+ + "partition {}", storagePartition, abortFailure);
+ }
+ }
+ }
} catch (Exception e) {
throw HyracksDataException.create(e);
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21550?usp=email
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I176d8077e4555069b5c183143eaab2bf03e019bd
Gerrit-Change-Number: 21550
Gerrit-PatchSet: 1
Gerrit-Owner: Ali Alsuliman <al...@gmail.com>