[XL] Change in asterixdb[master]: [ASTERIXDB-3771] Vector index DDL grammar and ANN top-k query support

[AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>]

From Ian Maxon <im...@apache.org>:

Attention is currently required from: Ali Alsuliman, Shahrzad Shirazi.

Ian Maxon has uploaded a new patch set (#27) to the change originally created by Hongyu Shi. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21287?usp=email )

The following approvals got outdated and were removed:
Integration-Tests+1 by Jenkins, Verified+1 by Jenkins


Change subject: [ASTERIXDB-3771] Vector index DDL grammar and ANN top-k query support
......................................................................

[ASTERIXDB-3771] Vector index DDL grammar and ANN top-k query support

- user model changes: yes
- storage format changes: no
- interface changes: yes

Add the SQL++ DDL/grammar for creating VTree vector indexes, plus the
query-side optimizer, physical operators, and runtime for approximate
nearest neighbor (ANN) top-k search over them.

DDL / grammar:
- SQLPP.jj: CREATE INDEX <name> ON <ds>(<field> VECTOR) TYPE VTREE
  WITH { dimension, similarity, num_clusters, train_list_fraction, ... }
- CreateIndexStatement: vector-index AST (VECTOR field spec, TYPE VTREE,
  INCLUDE fields, EXCLUDE UNKNOWN KEY)
- VectorIndexDeclUtil: validates the WITH-clause parameters (required
  similarity from an allowlist, dimension, quantization, etc.)

Query optimizer / physical:
- IntroduceTopKAccessMethodRule: matches LIMIT k -> ORDER BY ann_distance(...)
  and rewrites it to a vector index access
- VectorIndexAccessMethod: IAccessMethod for vector index selection
- VectorSearchPOperator / VectorJobGenParams: physical vector-search operator
- PushFilterIntoVectorSearchRule (+ VectorIndexFilterSchema/TypeEnvironment):
  pushes INCLUDE-field filters into the vector search

Runtime / metadata:
- ANNDistanceDescriptor: runtime evaluator for the ann_distance function
- VectorDistanceFunctionFactory / VectorSimilarityMetric: distance metric,
  fixed at index creation and carried by the factory
- MetadataProvider: builds the vector-search operator for the query path

Change-Id: Iaa502b57915c75921cd5aa5c62fc0e24261bb0f6
---
A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/VectorIndexFilterSchema.java
A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/VectorIndexFilterTypeEnvironment.java
A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/VectorSearchPOperator.java
M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/SqlppCompilationProvider.java
M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceSecondaryIndexInsertDeleteRule.java
A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushFilterIntoVectorSearchRule.java
M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SetAsterixPhysicalOperatorsRule.java
M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/AbstractIntroduceAccessMethodRule.java
M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/AccessMethodUtils.java
A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IntroduceTopKAccessMethodRule.java
A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/VectorIndexAccessMethod.java
A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/VectorJobGenParams.java
M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/util/ValidateUtil.java
A asterixdb/asterix-app/data/vector/glove-sub.adm
A asterixdb/asterix-app/data/vector/movie-sub.adm
M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/function/DatasetRewriter.java
M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/io/PersistedResourceRegistry.java
M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/NullMissingExceptionTest.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/NullMissingTest.java
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.1.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.2.update.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.3.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.4.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.5.plans.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.6.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.99.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-glove/create-index-vtree-glove.1.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-glove/create-index-vtree-glove.2.update.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-glove/create-index-vtree-glove.3.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-glove/create-index-vtree-glove.4.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-glove/create-index-vtree-glove.5.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-glove/create-index-vtree-glove.99.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.1.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.2.update.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.3.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.4.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.5.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.99.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-invalid-field-type/create-index-vtree-invalid-field-type.1.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-movie/create-index-vtree-movie.1.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-movie/create-index-vtree-movie.2.update.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-movie/create-index-vtree-movie.3.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-movie/create-index-vtree-movie.4.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-movie/create-index-vtree-movie.5.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-movie/create-index-vtree-movie.99.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree/create-index-vtree.1.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree/create-index-vtree.2.update.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree/create-index-vtree.3.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree/create-index-vtree.4.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree/create-index-vtree.5.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree/create-index-vtree.6.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree/create-index-vtree.7.plans.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree/create-index-vtree.99.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.5.regex
A asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.6.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-glove/create-index-vtree-glove.5.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.5.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-movie/create-index-vtree-movie.5.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree/create-index-vtree.5.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree/create-index-vtree.6.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree/create-index-vtree.7.regex
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/CompilerProperties.java
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/dataflow/DatasetLocalResource.java
M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/CreateIndexStatement.java
M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/CommonFunctionMapUtil.java
M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/FunctionUtil.java
A asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/VectorIndexDeclUtil.java
M asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj
M asterixdb/asterix-metadata/pom.xml
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/BTreeResourceFactoryProvider.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataManagerUtil.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Dataset.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/IndexTupleTranslator.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/DatasetUtil.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/KeyFieldTypeUtil.java
M asterixdb/asterix-om/src/main/java/org/apache/asterix/om/functions/BuiltinFunctions.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/evaluators/functions/vector/ANNDistanceDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/functions/FunctionCollection.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/base/PhysicalOperatorTag.java
83 files changed, 5,782 insertions(+), 59 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/87/21287/27
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21287?usp=email
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email

Gerrit-MessageType: newpatchset
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Iaa502b57915c75921cd5aa5c62fc0e24261bb0f6
Gerrit-Change-Number: 21287
Gerrit-PatchSet: 27
Gerrit-Owner: Hongyu Shi <le...@gmail.com>
Gerrit-Reviewer: Ali Alsuliman <al...@gmail.com>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Shahrzad Shirazi <sh...@ucr.edu>
Gerrit-Attention: Shahrzad Shirazi <sh...@ucr.edu>
Gerrit-Attention: Ali Alsuliman <al...@gmail.com>