[L] Change in asterixdb[master]: [ASTERIXDB-3885][STO] Compress VTree data components

[AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>]

From Ritik Raj <ri...@couchbase.com>:

Attention is currently required from: Ritik Raj.

Hello Anon. E. Moose #1000171, Jenkins, 

I'd like you to reexamine a change. Please visit

    https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21887?usp=email

to look at the new patch set (#4).

The following approvals got outdated and were removed:
Contrib-2 by Anon. E. Moose #1000171, Verified-1 by Jenkins


Change subject: [ASTERIXDB-3885][STO] Compress VTree data components
......................................................................

[ASTERIXDB-3885][STO] Compress VTree data components

- user model changes: yes
- storage format changes: no
- interface changes: no

Details:
- The VTree never received a compression scheme: its file manager passed
  null to the base, which resolves to NoOp, and built component
  references without the look-aside file. The loaders already drive the
  compressed page writer, so only the scheme and the file references
  were missing.
- Like a B-tree secondary, a VTree now takes a scheme of its own,
  recorded when it is created: the one its WITH clause names next to the
  vector parameters, or the configured default.

    CREATE INDEX i ON ds(emb VECTOR) TYPE VTREE
      WITH {"dimension": 384, "similarity": "cosine",
            "storage-block-compression": {"scheme": "none"}};

- Only .vct data components are compressed; the shared static structure
  is written once, outside flush and merge, and stays uncompressed.
- Each page's unused tail is zeroed before it is written. A written page
  is a reused buffer whose tail still held earlier data, and a VTree
  leaves much of it unused since every cluster ends on a partly filled
  page, so the tail compressed poorly and carried stale bytes.
- The scheme is persisted per index. A resource written before this has
  no key and reads back as NoOp, so existing indexes keep opening.
- Recovery resolves components through the scheme, finishes a delete
  that left a data file without its look-aside file (or the reverse),
  and removes the look-aside files of merged-away inputs.

Co-Authored-By: Claude Opus 5.5 <no...@anthropic.com>
Change-Id: I13f1ad75516025efb3a24a2d3715cdf519d27ae6
---
M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-compression-scheme/create-index-vtree-compression-scheme.1.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-compression-scheme/create-index-vtree-compression-scheme.2.update.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-compression-scheme/create-index-vtree-compression-scheme.3.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-compression-scheme/create-index-vtree-compression-scheme.4.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-compression-scheme/create-index-vtree-compression-scheme.5.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-compression-scheme/create-index-vtree-compression-scheme.99.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-invalid-compression-scheme/create-index-vtree-invalid-compression-scheme.1.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-compression-scheme/create-index-vtree-compression-scheme.5.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-metadata/create-index-vtree-metadata.5.adm
M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/CreateIndexStatement.java
M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/DatasetDeclParametersUtil.java
M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/VectorIndexDeclUtil.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/VTreeResourceFactoryProvider.java
M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/LSMVTreeLocalResource.java
M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/LSMVTreeLocalResourceFactory.java
M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/LSMVTreeFileManager.java
M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/utils/LSMVTreeUtils.java
M hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/frames/VTreeNSMFrame.java
M hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeBulkLoader.java
M hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeFlushLoader.java
M hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeStaticStructureBuilder.java
A hyracks-fullstack/hyracks/hyracks-tests/hyracks-storage-am-lsm-vtree-test/src/test/java/org/apache/hyracks/storage/am/lsm/vector/LSMVTreeCompressedMergeTest.java
M hyracks-fullstack/hyracks/hyracks-tests/hyracks-storage-am-lsm-vtree-test/src/test/java/org/apache/hyracks/storage/am/lsm/vector/LSMVTreeFileManagerCrashRecoveryTest.java
M hyracks-fullstack/hyracks/hyracks-tests/hyracks-storage-am-lsm-vtree-test/src/test/java/org/apache/hyracks/storage/am/lsm/vector/LSMVTreeLocalResourceJsonTest.java
M hyracks-fullstack/hyracks/hyracks-tests/hyracks-storage-am-lsm-vtree-test/src/test/java/org/apache/hyracks/storage/am/lsm/vector/LSMVTreeMergeTest.java
M hyracks-fullstack/hyracks/hyracks-tests/hyracks-storage-am-lsm-vtree-test/src/test/java/org/apache/hyracks/storage/am/lsm/vector/util/LSMVTreeTestContext.java
28 files changed, 599 insertions(+), 38 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/87/21887/4
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21887?usp=email
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email

Gerrit-MessageType: newpatchset
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I13f1ad75516025efb3a24a2d3715cdf519d27ae6
Gerrit-Change-Number: 21887
Gerrit-PatchSet: 4
Gerrit-Owner: Ritik Raj <ri...@couchbase.com>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Attention: Ritik Raj <ri...@couchbase.com>