[L] Change in asterixdb[master]: [ASTERIXDB-3760][ASTERIXDB-3754][RT][STO] Handlers of vtree build on ...

[AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>]

From Hongyu Shi <le...@gmail.com>:

Attention is currently required from: Hongyu Shi.

Hello Anon. E. Moose #1000171, Jenkins, 

I'd like you to reexamine a change. Please visit

    https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21535?usp=email

to look at the new patch set (#6).


Change subject: [ASTERIXDB-3760][ASTERIXDB-3754][RT][STO] Handlers of vtree build on dimension mismatch
......................................................................

[ASTERIXDB-3760][ASTERIXDB-3754][RT][STO] Handlers of vtree build on dimension mismatch

- user model changes: yes
- storage format changes: no
- interface changes: yes

Details:
By design, a vector index builds only over the embeddings whose
dimension matches the DDL and skips the rest, so a build is implicitly
partial. Each stage applied that rule on its own, which left the runtime
exception handling tangled: before this change, when no vector matched,
CREATE INDEX reported success while the bulk load dropped every record
and ANN queries on the finished index returned no rows; when only some
matched, training ran on vectors the index would never hold, one could
become a centroid, their components shaped the quantization constants,
and a vector longer than the declared dimension threw
ArrayIndexOutOfBoundsException in the distance loop.

Fix: apply the rule at every stage that consumes a sampled vector, and
fail the build when nothing is left to train on.

- Training: seed k-means from the first indexable vector, and fail when
  the sample held vectors but none are indexable.
- Quantization: skip non-indexable vectors so the constants describe what
  the index will hold, and fail the same way one job earlier.
- Reporting: the static-structure creator no longer builds after an
  upstream failure, which replaced the real error with "no training
  vectors were found"; the empty quantization payload names its causes;
  both jobs share one message.
- Metadata: drop the 384 fallback for a missing `dimension`.
- Storage: bind metaFrame up front so end() works for an empty load.

Tests: operator coverage for both guards and for the partial and empty
cases, and a runtimets golden for the DDL error.

Ext-ref: MB-73219
Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
Change-Id: I44e13a38e10e09d0c439063582b2747942476f26
---
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-dimension-mismatch/create-index-vtree-dimension-mismatch.1.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-dimension-mismatch/create-index-vtree-dimension-mismatch.2.update.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-dimension-mismatch/create-index-vtree-dimension-mismatch.3.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-dimension-mismatch/create-index-vtree-dimension-mismatch.4.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-dimension-mismatch/create-index-vtree-dimension-mismatch.99.ddl.sqlpp
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryVectorOperationsHelper.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/VTreeResourceFactoryProvider.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/HierarchicalKMeansPlusPlusCentroidsOperatorDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/VTreeStaticStructureCreatorOperatorDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/VectorComponentExtractorOperatorDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/VectorIndexBuildErrors.java
M asterixdb/asterix-runtime/src/test/java/org/apache/asterix/runtime/operators/HierarchicalKMeansPlusPlusCentroidsOperatorDescriptorTest.java
M asterixdb/asterix-runtime/src/test/java/org/apache/asterix/runtime/operators/VectorComponentExtractorOperatorDescriptorTest.java
M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/QuantizedIndexCreateOperatorDescriptor.java
M hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeBulkLoader.java
16 files changed, 483 insertions(+), 20 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/35/21535/6
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21535?usp=email
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email

Gerrit-MessageType: newpatchset
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I44e13a38e10e09d0c439063582b2747942476f26
Gerrit-Change-Number: 21535
Gerrit-PatchSet: 6
Gerrit-Owner: Hongyu Shi <le...@gmail.com>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Attention: Hongyu Shi <le...@gmail.com>