[L] Change in asterixdb[master]: [ASTERIXDB-3760][RT][STO] Handlers of vtree build on dimension mismatch

[AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>]

From Hongyu Shi <le...@gmail.com>:

Attention is currently required from: Hongyu Shi.

Hello Anon. E. Moose #1000171, Ian Maxon, Jenkins, 

I'd like you to reexamine a change. Please visit

    https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21535?usp=email

to look at the new patch set (#7).

The following approvals got outdated and were removed:
Contrib-2 by Anon. E. Moose #1000171, Integration-Tests+1 by Jenkins, Verified+1 by Jenkins


Change subject: [ASTERIXDB-3760][RT][STO] Handlers of vtree build on dimension mismatch
......................................................................

[ASTERIXDB-3760][RT][STO] Handlers of vtree build on dimension mismatch

- user model changes: yes
- storage format changes: no
- interface changes: yes

Details:
By design, a vector index builds only over the embeddings whose
dimension matches the DDL and skips the rest, so a build is implicitly
partial. Each stage applied that rule on its own, which left the runtime
exception handling tangled: before this change, when no vector matched,
CREATE INDEX reported success while the bulk load dropped every record
and ANN queries on the finished index returned no rows; when only some
matched, training ran on vectors the index would never hold, one could
become a centroid, their components shaped the quantization constants,
and a vector longer than the declared dimension threw
ArrayIndexOutOfBoundsException in the distance loop.

Fix: apply the rule at every stage that consumes a sampled vector, and
reject a build that would leave a partition with nothing to index. The
rejection is raised in the first job, before any training work, and
names both the declared and the observed dimension.

- Training: test the dimension wherever a sampled vector is consumed,
  and seed k-means from the first indexable one.
- Quantization: skip non-indexable vectors so the constants describe the
  vectors the index will actually hold, and reject a partition whose
  whole sample is non-indexable.
- Reporting: the static-structure creator no longer builds after an
  upstream failure, which replaced the real error with "no training
  vectors were found"; the empty quantization payload now names its
  likely causes instead of a byte count.
- Metadata: drop the 384 fallback for a missing `dimension`.
- Storage: bind metaFrame in end(), where takeBlock needs it, so an
  empty load works. takeBlock reads getMaxPage() without binding the
  frame, unlike takePage.

Tests: operator coverage that off-dimension vectors do not shape the
centroids or the constants, for the partial and empty cases, and a
runtimets golden for the DDL error.

Ext-ref: MB-73219
Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
Change-Id: I44e13a38e10e09d0c439063582b2747942476f26
---
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-dimension-mismatch/create-index-vtree-dimension-mismatch.1.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-dimension-mismatch/create-index-vtree-dimension-mismatch.2.update.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-dimension-mismatch/create-index-vtree-dimension-mismatch.3.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-dimension-mismatch/create-index-vtree-dimension-mismatch.4.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-dimension-mismatch/create-index-vtree-dimension-mismatch.99.ddl.sqlpp
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryVectorOperationsHelper.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/VTreeResourceFactoryProvider.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/HierarchicalKMeansPlusPlusCentroidsOperatorDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/VTreeStaticStructureCreatorOperatorDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/VectorComponentExtractorOperatorDescriptor.java
M asterixdb/asterix-runtime/src/test/java/org/apache/asterix/runtime/operators/HierarchicalKMeansPlusPlusCentroidsOperatorDescriptorTest.java
M asterixdb/asterix-runtime/src/test/java/org/apache/asterix/runtime/operators/VectorComponentExtractorOperatorDescriptorTest.java
M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/QuantizedIndexCreateOperatorDescriptor.java
M hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeBulkLoader.java
15 files changed, 406 insertions(+), 20 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/35/21535/7
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21535?usp=email
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email

Gerrit-MessageType: newpatchset
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I44e13a38e10e09d0c439063582b2747942476f26
Gerrit-Change-Number: 21535
Gerrit-PatchSet: 7
Gerrit-Owner: Hongyu Shi <le...@gmail.com>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Attention: Hongyu Shi <le...@gmail.com>