You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@asterixdb.apache.org by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu> on 2026/09/01 13:17:31 UTC
[XL] Change in asterixdb[master]: [ASTERIXDB-3817][COMP][MTD] Infer vector index knobs from cardinality
From Hongyu Shi <le...@gmail.com>:
Hongyu Shi has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21652?usp=email )
Change subject: [ASTERIXDB-3817][COMP][MTD] Infer vector index knobs from cardinality
......................................................................
[ASTERIXDB-3817][COMP][MTD] Infer vector index knobs from cardinality
- user model changes: yes
- storage format changes: no
- interface changes: yes
Details:
A vector index needs two size knobs the user is not expected to supply: the leaf
cluster count and the training list the k-means sees. num_clusters defaulted to
sqrt(D). On a collection small enough that sqrt(D) leaves too few rows per
cluster there is not enough data to train on. It also reached the index record
only when the user named it. What a build actually used was therefore
unrecoverable afterwards. num_clusters and train_list_fraction were not
correlated with each other either. Neither was inferred from the dataset
cardinality per storage partition.
VectorIndexSizeInference derives both from the per-partition cardinality D. That
is the unit a build works in because each partition trains its own tree.
num_clusters becomes sqrt(D) and falls back to ceil(D/100) on a small
collection. The advised train list size is 50 records per cluster. A
user-supplied value is checked against the same D. What the build cannot work
with is an error and what is merely unusual warns. Inference runs on the DDL
path and the result is persisted so the record states what the build used. That
also moves the existing missing-ANALYZE and empty-collection checks from job
build to compile time.
Ext-ref: MB-73665, MB-73194
Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
Change-Id: I5d90fdd079f0931edaeb46a26ac2c717e07a043c
---
M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.01.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.02.update.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.03.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.04.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.99.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.01.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.02.update.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.03.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.04.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.05.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.99.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.01.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.02.update.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.03.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.04.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.05.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.99.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.01.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.02.update.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.03.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.04.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.99.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.05.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.05.adm
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/VectorIndexDeclUtil.java
M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryVectorOperationsHelper.java
A asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/VectorIndexSizeInference.java
M asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/VectorIndexParametersTupleTranslatorTest.java
A asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/utils/VectorIndexSizeInferenceTest.java
M asterixdb/asterix-om/src/main/java/org/apache/asterix/om/vector/VectorIndexParameters.java
M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/QuantizedIndexCreateOperatorDescriptor.java
35 files changed, 1,211 insertions(+), 52 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/52/21652/1
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
index a7a42b5..4061407 100644
--- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
@@ -238,6 +238,7 @@
import org.apache.asterix.metadata.utils.KeyFieldTypeUtil;
import org.apache.asterix.metadata.utils.SampleOperationsHelper;
import org.apache.asterix.metadata.utils.TypeUtil;
+import org.apache.asterix.metadata.utils.VectorIndexSizeInference;
import org.apache.asterix.om.base.ANull;
import org.apache.asterix.om.base.IAObject;
import org.apache.asterix.om.types.AOrderedListType;
@@ -250,6 +251,7 @@
import org.apache.asterix.om.types.TypeSignature;
import org.apache.asterix.om.types.hierachy.ATypeHierarchy;
import org.apache.asterix.om.utils.RecordUtil;
+import org.apache.asterix.om.vector.VectorIndexParameters;
import org.apache.asterix.optimizer.rules.visitor.FunctionCardinalityInferenceVisitor;
import org.apache.asterix.runtime.fulltext.AbstractFullTextFilterDescriptor;
import org.apache.asterix.runtime.fulltext.FullTextConfigDescriptor;
@@ -1914,9 +1916,15 @@
// excludeUnknownKey is a BTree/Array-index DDL option that does not apply to a vector index;
// the vector CREATE INDEX path never sets it, so it is always empty here.
- indexDetails = new Index.VectorIndexDetails(keyFieldNames, includeFieldNames,
- includeFieldSourceIndicators, includeFieldTypes, false, OptionalBoolean.empty(),
- stmtCreateIndex.getVectorParameters());
+ // Fix the sizing before the record is written. num_clusters and train_list_fraction are
+ // derived from the per-partition cardinality when the user leaves them out, and the record is
+ // written before any build job runs, so deriving them later would leave the catalog silent
+ // about what the build actually used. Same reason the seed is drawn at DDL time.
+ VectorIndexParameters vectorParameters = VectorIndexSizeInference.infer(metadataProvider, ds,
+ stmtCreateIndex.getVectorParameters(), warningCollector, sourceLoc);
+ indexDetails =
+ new Index.VectorIndexDetails(keyFieldNames, includeFieldNames, includeFieldSourceIndicators,
+ includeFieldTypes, false, OptionalBoolean.empty(), vectorParameters);
Index newIndex = new Index(databaseName, dataverseName, datasetName, indexName, indexType, indexDetails,
stmtCreateIndex.isEnforced(), false, MetadataUtil.PENDING_ADD_OP, creator);
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml
index bfe36c6..56ef204 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml
@@ -80,6 +80,28 @@
</compilation-unit>
</test-case>
<test-case FilePath="vector">
+ <compilation-unit name="create-index-vtree-clusters-not-recommended">
+ <output-dir compare="Text">create-index-vtree-clusters-not-recommended</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="vector">
+ <compilation-unit name="create-index-vtree-clusters-exceed-rows">
+ <output-dir compare="Text">create-index-vtree-clusters-exceed-rows</output-dir>
+ <expected-error>records per storage partition available to train on</expected-error>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="vector">
+ <compilation-unit name="create-index-vtree-train-list-too-small">
+ <output-dir compare="Text">create-index-vtree-train-list-too-small</output-dir>
+ <expected-error>fewer than the 2 clusters to be built</expected-error>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="vector">
+ <compilation-unit name="create-index-vtree-derived-sizing">
+ <output-dir compare="Text">create-index-vtree-derived-sizing</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="vector">
<compilation-unit name="create-index-vtree-swapped-args">
<output-dir compare="Text">create-index-vtree-swapped-args</output-dir>
</compilation-unit>
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.01.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.01.ddl.sqlpp
new file mode 100644
index 0000000..2497935
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.01.ddl.sqlpp
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// TYPE VTREE is optional when the indexed field carries the VECTOR annotation: no other index type
+// accepts one, so the annotation alone determines the type. Everything below is deliberately the
+// same shape as create-index-vtree, minus the TYPE clause.
+DROP DATAVERSE test IF EXISTS;
+CREATE DATAVERSE test;
+USE test;
+CREATE TYPE MovieType AS {
+ id: int,
+ embedding: [double]
+};
+CREATE DATASET MovieSmall(MovieType) PRIMARY KEY id;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.02.update.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.02.update.sqlpp
new file mode 100644
index 0000000..94517b1
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.02.update.sqlpp
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+USE test;
+
+// The 4th component is the only thing that varies within each block, so the 3 nearest to the
+// origin are ids 1, 2 and 3 regardless of how the two blocks are clustered.
+INSERT INTO MovieSmall ([
+ {"id": 1, "embedding": [0.0, 0.0, 0.0, 0.1]},
+ {"id": 2, "embedding": [0.0, 0.0, 0.0, 0.2]},
+ {"id": 3, "embedding": [0.0, 0.0, 0.0, 0.3]},
+ {"id": 4, "embedding": [0.0, 0.0, 0.0, 0.4]},
+ {"id": 5, "embedding": [0.0, 0.0, 0.0, 0.5]},
+ {"id": 6, "embedding": [0.0, 0.0, 0.0, 0.6]},
+ {"id": 7, "embedding": [0.0, 0.0, 0.0, 0.7]},
+ {"id": 8, "embedding": [0.0, 0.0, 0.0, 0.8]},
+ {"id": 9, "embedding": [0.0, 0.0, 0.0, 0.9]},
+ {"id": 10, "embedding": [0.0, 0.0, 0.0, 1.0]},
+ {"id": 11, "embedding": [0.0, 0.0, 0.0, 1.1]},
+ {"id": 12, "embedding": [0.0, 0.0, 0.0, 1.2]},
+ {"id": 13, "embedding": [0.0, 0.0, 0.0, 1.3]},
+ {"id": 14, "embedding": [0.0, 0.0, 0.0, 1.4]},
+ {"id": 15, "embedding": [0.0, 0.0, 0.0, 1.5]},
+ {"id": 16, "embedding": [10.0, 10.0, 10.0, 0.1]},
+ {"id": 17, "embedding": [10.0, 10.0, 10.0, 0.2]},
+ {"id": 18, "embedding": [10.0, 10.0, 10.0, 0.3]},
+ {"id": 19, "embedding": [10.0, 10.0, 10.0, 0.4]},
+ {"id": 20, "embedding": [10.0, 10.0, 10.0, 0.5]},
+ {"id": 21, "embedding": [10.0, 10.0, 10.0, 0.6]},
+ {"id": 22, "embedding": [10.0, 10.0, 10.0, 0.7]},
+ {"id": 23, "embedding": [10.0, 10.0, 10.0, 0.8]},
+ {"id": 24, "embedding": [10.0, 10.0, 10.0, 0.9]},
+ {"id": 25, "embedding": [10.0, 10.0, 10.0, 1.0]},
+ {"id": 26, "embedding": [10.0, 10.0, 10.0, 1.1]},
+ {"id": 27, "embedding": [10.0, 10.0, 10.0, 1.2]},
+ {"id": 28, "embedding": [10.0, 10.0, 10.0, 1.3]},
+ {"id": 29, "embedding": [10.0, 10.0, 10.0, 1.4]},
+ {"id": 30, "embedding": [10.0, 10.0, 10.0, 1.5]}
+]);
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.03.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.03.ddl.sqlpp
new file mode 100644
index 0000000..b63fc7a
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.03.ddl.sqlpp
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// A sample index is required before creating a vector index (quantization training scans it).
+ANALYZE DATASET test.MovieSmall WITH {"sample-seed": 1000};
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.04.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.04.ddl.sqlpp
new file mode 100644
index 0000000..10cf5c8
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.04.ddl.sqlpp
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+USE test;
+
+// 30 records, so no storage partition holds anywhere near 5000 of them. More clusters than rows cannot
+// be built: the shortfall would be padded with perturbed duplicates of real centroids.
+CREATE INDEX idx_emb
+ ON MovieSmall(embedding VECTOR)
+ TYPE VTREE
+ WITH { "dimension": 4, "similarity": "euclidean", "num_clusters": 5000, "seed": 42 }
+ EXCLUDE UNKNOWN KEY;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.99.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.99.ddl.sqlpp
new file mode 100644
index 0000000..d240ef2
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-exceed-rows/create-index-vtree-clusters-exceed-rows.99.ddl.sqlpp
@@ -0,0 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+DROP DATAVERSE test IF EXISTS;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.01.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.01.ddl.sqlpp
new file mode 100644
index 0000000..2497935
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.01.ddl.sqlpp
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// TYPE VTREE is optional when the indexed field carries the VECTOR annotation: no other index type
+// accepts one, so the annotation alone determines the type. Everything below is deliberately the
+// same shape as create-index-vtree, minus the TYPE clause.
+DROP DATAVERSE test IF EXISTS;
+CREATE DATAVERSE test;
+USE test;
+CREATE TYPE MovieType AS {
+ id: int,
+ embedding: [double]
+};
+CREATE DATASET MovieSmall(MovieType) PRIMARY KEY id;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.02.update.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.02.update.sqlpp
new file mode 100644
index 0000000..94517b1
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.02.update.sqlpp
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+USE test;
+
+// The 4th component is the only thing that varies within each block, so the 3 nearest to the
+// origin are ids 1, 2 and 3 regardless of how the two blocks are clustered.
+INSERT INTO MovieSmall ([
+ {"id": 1, "embedding": [0.0, 0.0, 0.0, 0.1]},
+ {"id": 2, "embedding": [0.0, 0.0, 0.0, 0.2]},
+ {"id": 3, "embedding": [0.0, 0.0, 0.0, 0.3]},
+ {"id": 4, "embedding": [0.0, 0.0, 0.0, 0.4]},
+ {"id": 5, "embedding": [0.0, 0.0, 0.0, 0.5]},
+ {"id": 6, "embedding": [0.0, 0.0, 0.0, 0.6]},
+ {"id": 7, "embedding": [0.0, 0.0, 0.0, 0.7]},
+ {"id": 8, "embedding": [0.0, 0.0, 0.0, 0.8]},
+ {"id": 9, "embedding": [0.0, 0.0, 0.0, 0.9]},
+ {"id": 10, "embedding": [0.0, 0.0, 0.0, 1.0]},
+ {"id": 11, "embedding": [0.0, 0.0, 0.0, 1.1]},
+ {"id": 12, "embedding": [0.0, 0.0, 0.0, 1.2]},
+ {"id": 13, "embedding": [0.0, 0.0, 0.0, 1.3]},
+ {"id": 14, "embedding": [0.0, 0.0, 0.0, 1.4]},
+ {"id": 15, "embedding": [0.0, 0.0, 0.0, 1.5]},
+ {"id": 16, "embedding": [10.0, 10.0, 10.0, 0.1]},
+ {"id": 17, "embedding": [10.0, 10.0, 10.0, 0.2]},
+ {"id": 18, "embedding": [10.0, 10.0, 10.0, 0.3]},
+ {"id": 19, "embedding": [10.0, 10.0, 10.0, 0.4]},
+ {"id": 20, "embedding": [10.0, 10.0, 10.0, 0.5]},
+ {"id": 21, "embedding": [10.0, 10.0, 10.0, 0.6]},
+ {"id": 22, "embedding": [10.0, 10.0, 10.0, 0.7]},
+ {"id": 23, "embedding": [10.0, 10.0, 10.0, 0.8]},
+ {"id": 24, "embedding": [10.0, 10.0, 10.0, 0.9]},
+ {"id": 25, "embedding": [10.0, 10.0, 10.0, 1.0]},
+ {"id": 26, "embedding": [10.0, 10.0, 10.0, 1.1]},
+ {"id": 27, "embedding": [10.0, 10.0, 10.0, 1.2]},
+ {"id": 28, "embedding": [10.0, 10.0, 10.0, 1.3]},
+ {"id": 29, "embedding": [10.0, 10.0, 10.0, 1.4]},
+ {"id": 30, "embedding": [10.0, 10.0, 10.0, 1.5]}
+]);
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.03.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.03.ddl.sqlpp
new file mode 100644
index 0000000..b63fc7a
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.03.ddl.sqlpp
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// A sample index is required before creating a vector index (quantization training scans it).
+ANALYZE DATASET test.MovieSmall WITH {"sample-seed": 1000};
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.04.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.04.ddl.sqlpp
new file mode 100644
index 0000000..ca7d0d3
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.04.ddl.sqlpp
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+USE test;
+
+// 30 records, so the recommendation is 1 cluster whatever the partition count: the small-data rule is
+// ceil(D/100). Asking for 3 is legal and is built, but it is far enough from the recommendation to be
+// worth saying so, and it warns. The warning's own conditions and text are covered by
+// VectorIndexSizingTest; what this case pins is that the value is accepted and built, not rejected.
+CREATE INDEX idx_emb
+ ON MovieSmall(embedding VECTOR)
+ TYPE VTREE
+ WITH { "dimension": 4, "similarity": "euclidean", "num_clusters": 3, "seed": 42 }
+ EXCLUDE UNKNOWN KEY;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.05.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.05.query.sqlpp
new file mode 100644
index 0000000..b670889
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.05.query.sqlpp
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// The index is built despite the warning, and the value the user asked for is what got recorded.
+SELECT i.num_clusters
+FROM Metadata.`Index` i
+WHERE i.DataverseName = "test" AND i.DatasetName = "MovieSmall" AND i.IndexName = "idx_emb";
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.99.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.99.ddl.sqlpp
new file mode 100644
index 0000000..d240ef2
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.99.ddl.sqlpp
@@ -0,0 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+DROP DATAVERSE test IF EXISTS;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.01.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.01.ddl.sqlpp
new file mode 100644
index 0000000..2497935
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.01.ddl.sqlpp
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// TYPE VTREE is optional when the indexed field carries the VECTOR annotation: no other index type
+// accepts one, so the annotation alone determines the type. Everything below is deliberately the
+// same shape as create-index-vtree, minus the TYPE clause.
+DROP DATAVERSE test IF EXISTS;
+CREATE DATAVERSE test;
+USE test;
+CREATE TYPE MovieType AS {
+ id: int,
+ embedding: [double]
+};
+CREATE DATASET MovieSmall(MovieType) PRIMARY KEY id;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.02.update.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.02.update.sqlpp
new file mode 100644
index 0000000..94517b1
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.02.update.sqlpp
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+USE test;
+
+// The 4th component is the only thing that varies within each block, so the 3 nearest to the
+// origin are ids 1, 2 and 3 regardless of how the two blocks are clustered.
+INSERT INTO MovieSmall ([
+ {"id": 1, "embedding": [0.0, 0.0, 0.0, 0.1]},
+ {"id": 2, "embedding": [0.0, 0.0, 0.0, 0.2]},
+ {"id": 3, "embedding": [0.0, 0.0, 0.0, 0.3]},
+ {"id": 4, "embedding": [0.0, 0.0, 0.0, 0.4]},
+ {"id": 5, "embedding": [0.0, 0.0, 0.0, 0.5]},
+ {"id": 6, "embedding": [0.0, 0.0, 0.0, 0.6]},
+ {"id": 7, "embedding": [0.0, 0.0, 0.0, 0.7]},
+ {"id": 8, "embedding": [0.0, 0.0, 0.0, 0.8]},
+ {"id": 9, "embedding": [0.0, 0.0, 0.0, 0.9]},
+ {"id": 10, "embedding": [0.0, 0.0, 0.0, 1.0]},
+ {"id": 11, "embedding": [0.0, 0.0, 0.0, 1.1]},
+ {"id": 12, "embedding": [0.0, 0.0, 0.0, 1.2]},
+ {"id": 13, "embedding": [0.0, 0.0, 0.0, 1.3]},
+ {"id": 14, "embedding": [0.0, 0.0, 0.0, 1.4]},
+ {"id": 15, "embedding": [0.0, 0.0, 0.0, 1.5]},
+ {"id": 16, "embedding": [10.0, 10.0, 10.0, 0.1]},
+ {"id": 17, "embedding": [10.0, 10.0, 10.0, 0.2]},
+ {"id": 18, "embedding": [10.0, 10.0, 10.0, 0.3]},
+ {"id": 19, "embedding": [10.0, 10.0, 10.0, 0.4]},
+ {"id": 20, "embedding": [10.0, 10.0, 10.0, 0.5]},
+ {"id": 21, "embedding": [10.0, 10.0, 10.0, 0.6]},
+ {"id": 22, "embedding": [10.0, 10.0, 10.0, 0.7]},
+ {"id": 23, "embedding": [10.0, 10.0, 10.0, 0.8]},
+ {"id": 24, "embedding": [10.0, 10.0, 10.0, 0.9]},
+ {"id": 25, "embedding": [10.0, 10.0, 10.0, 1.0]},
+ {"id": 26, "embedding": [10.0, 10.0, 10.0, 1.1]},
+ {"id": 27, "embedding": [10.0, 10.0, 10.0, 1.2]},
+ {"id": 28, "embedding": [10.0, 10.0, 10.0, 1.3]},
+ {"id": 29, "embedding": [10.0, 10.0, 10.0, 1.4]},
+ {"id": 30, "embedding": [10.0, 10.0, 10.0, 1.5]}
+]);
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.03.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.03.ddl.sqlpp
new file mode 100644
index 0000000..b63fc7a
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.03.ddl.sqlpp
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// A sample index is required before creating a vector index (quantization training scans it).
+ANALYZE DATASET test.MovieSmall WITH {"sample-seed": 1000};
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.04.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.04.ddl.sqlpp
new file mode 100644
index 0000000..ff1c9b3
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.04.ddl.sqlpp
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+USE test;
+
+// Neither num_clusters nor train_list_fraction: both are derived from the per-partition cardinality.
+CREATE INDEX idx_emb
+ ON MovieSmall(embedding VECTOR)
+ TYPE VTREE
+ WITH { "dimension": 4, "similarity": "euclidean", "seed": 42 }
+ EXCLUDE UNKNOWN KEY;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.05.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.05.query.sqlpp
new file mode 100644
index 0000000..8d2b433
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.05.query.sqlpp
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// Both derived values must reach the catalog. Asserting presence rather than the numbers themselves:
+// they follow from the per-partition cardinality, so pinning them here would pin the suite's partition
+// count too. The arithmetic is covered exactly by VectorIndexSizingTest.
+SELECT i.num_clusters IS NOT MISSING AS records_num_clusters,
+ i.train_list_fraction IS NOT MISSING AS records_train_list_fraction,
+ i.num_clusters >= 1 AS num_clusters_is_positive,
+ i.train_list_fraction > 0 AND i.train_list_fraction <= 1 AS fraction_in_range
+FROM Metadata.`Index` i
+WHERE i.DataverseName = "test" AND i.DatasetName = "MovieSmall" AND i.IndexName = "idx_emb";
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.99.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.99.ddl.sqlpp
new file mode 100644
index 0000000..d240ef2
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.99.ddl.sqlpp
@@ -0,0 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+DROP DATAVERSE test IF EXISTS;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.01.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.01.ddl.sqlpp
new file mode 100644
index 0000000..2497935
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.01.ddl.sqlpp
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// TYPE VTREE is optional when the indexed field carries the VECTOR annotation: no other index type
+// accepts one, so the annotation alone determines the type. Everything below is deliberately the
+// same shape as create-index-vtree, minus the TYPE clause.
+DROP DATAVERSE test IF EXISTS;
+CREATE DATAVERSE test;
+USE test;
+CREATE TYPE MovieType AS {
+ id: int,
+ embedding: [double]
+};
+CREATE DATASET MovieSmall(MovieType) PRIMARY KEY id;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.02.update.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.02.update.sqlpp
new file mode 100644
index 0000000..94517b1
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.02.update.sqlpp
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+USE test;
+
+// The 4th component is the only thing that varies within each block, so the 3 nearest to the
+// origin are ids 1, 2 and 3 regardless of how the two blocks are clustered.
+INSERT INTO MovieSmall ([
+ {"id": 1, "embedding": [0.0, 0.0, 0.0, 0.1]},
+ {"id": 2, "embedding": [0.0, 0.0, 0.0, 0.2]},
+ {"id": 3, "embedding": [0.0, 0.0, 0.0, 0.3]},
+ {"id": 4, "embedding": [0.0, 0.0, 0.0, 0.4]},
+ {"id": 5, "embedding": [0.0, 0.0, 0.0, 0.5]},
+ {"id": 6, "embedding": [0.0, 0.0, 0.0, 0.6]},
+ {"id": 7, "embedding": [0.0, 0.0, 0.0, 0.7]},
+ {"id": 8, "embedding": [0.0, 0.0, 0.0, 0.8]},
+ {"id": 9, "embedding": [0.0, 0.0, 0.0, 0.9]},
+ {"id": 10, "embedding": [0.0, 0.0, 0.0, 1.0]},
+ {"id": 11, "embedding": [0.0, 0.0, 0.0, 1.1]},
+ {"id": 12, "embedding": [0.0, 0.0, 0.0, 1.2]},
+ {"id": 13, "embedding": [0.0, 0.0, 0.0, 1.3]},
+ {"id": 14, "embedding": [0.0, 0.0, 0.0, 1.4]},
+ {"id": 15, "embedding": [0.0, 0.0, 0.0, 1.5]},
+ {"id": 16, "embedding": [10.0, 10.0, 10.0, 0.1]},
+ {"id": 17, "embedding": [10.0, 10.0, 10.0, 0.2]},
+ {"id": 18, "embedding": [10.0, 10.0, 10.0, 0.3]},
+ {"id": 19, "embedding": [10.0, 10.0, 10.0, 0.4]},
+ {"id": 20, "embedding": [10.0, 10.0, 10.0, 0.5]},
+ {"id": 21, "embedding": [10.0, 10.0, 10.0, 0.6]},
+ {"id": 22, "embedding": [10.0, 10.0, 10.0, 0.7]},
+ {"id": 23, "embedding": [10.0, 10.0, 10.0, 0.8]},
+ {"id": 24, "embedding": [10.0, 10.0, 10.0, 0.9]},
+ {"id": 25, "embedding": [10.0, 10.0, 10.0, 1.0]},
+ {"id": 26, "embedding": [10.0, 10.0, 10.0, 1.1]},
+ {"id": 27, "embedding": [10.0, 10.0, 10.0, 1.2]},
+ {"id": 28, "embedding": [10.0, 10.0, 10.0, 1.3]},
+ {"id": 29, "embedding": [10.0, 10.0, 10.0, 1.4]},
+ {"id": 30, "embedding": [10.0, 10.0, 10.0, 1.5]}
+]);
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.03.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.03.ddl.sqlpp
new file mode 100644
index 0000000..b63fc7a
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.03.ddl.sqlpp
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+// A sample index is required before creating a vector index (quantization training scans it).
+ANALYZE DATASET test.MovieSmall WITH {"sample-seed": 1000};
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.04.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.04.ddl.sqlpp
new file mode 100644
index 0000000..01e1a77
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.04.ddl.sqlpp
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+USE test;
+
+// A fraction this small leaves fewer training records than clusters, so there is not even one record
+// per centroid to place. Kept at 2 clusters so the collection is never smaller than the cluster count
+// itself -- D here is 30 records spread over the suite's storage partitions, so it is single digits.
+CREATE INDEX idx_emb
+ ON MovieSmall(embedding VECTOR)
+ TYPE VTREE
+ WITH { "dimension": 4, "similarity": "euclidean", "num_clusters": 2, "train_list_fraction": 0.01, "seed": 42 }
+ EXCLUDE UNKNOWN KEY;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.99.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.99.ddl.sqlpp
new file mode 100644
index 0000000..d240ef2
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-train-list-too-small/create-index-vtree-train-list-too-small.99.ddl.sqlpp
@@ -0,0 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+DROP DATAVERSE test IF EXISTS;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.05.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.05.adm
new file mode 100644
index 0000000..d22af83
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-clusters-not-recommended/create-index-vtree-clusters-not-recommended.05.adm
@@ -0,0 +1 @@
+{ "num_clusters": 3 }
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.05.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.05.adm
new file mode 100644
index 0000000..03552f2
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-derived-sizing/create-index-vtree-derived-sizing.05.adm
@@ -0,0 +1 @@
+{ "records_num_clusters": true, "records_train_list_fraction": true, "num_clusters_is_positive": true, "fraction_in_range": true }
diff --git a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
index fdfa61f..548d6ba 100644
--- a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
+++ b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
@@ -371,6 +371,8 @@
UNSUPPORTED_MULTIPLE_STATEMENTS_FOR_DELIVERY_MODE(1259),
UNSUPPORTED_VECTOR_ELEMENT_TYPE(1260),
VECTOR_DIMENSION_MISMATCH(1261),
+ VECTOR_INDEX_NUM_CLUSTERS_NOT_RECOMMENDED(1262),
+ VECTOR_INDEX_TRAIN_LIST_TOO_SMALL(1263),
// Feed errors
DATAFLOW_ILLEGAL_STATE(3001),
diff --git a/asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties b/asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
index 0ff4d5a..292a7e2 100644
--- a/asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
+++ b/asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
@@ -373,6 +373,8 @@
1259 = %1$s execution is not supported for a request with multiple statements
1260 = Unsupported type %1$s in the vector value; the supported element types are numbers
1261 = Vector dimension mismatch: expected %1$s dimensions, but got %2$s
+1262 = Vector index num_clusters %1$s is outside the recommended range for this collection; %2$s is recommended for %3$s records per storage partition
+1263 = Vector index train_list_fraction %1$s trains on %2$s records per storage partition, fewer than the %3$s recommended for %4$s clusters; raise it to at least %5$s
# Feed Errors
3001 = Illegal state.
diff --git a/asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/VectorIndexDeclUtil.java b/asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/VectorIndexDeclUtil.java
index 9b12e72..19b9769 100644
--- a/asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/VectorIndexDeclUtil.java
+++ b/asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/VectorIndexDeclUtil.java
@@ -32,6 +32,7 @@
import java.util.Arrays;
import java.util.Locale;
+import java.util.OptionalDouble;
import java.util.OptionalInt;
import java.util.concurrent.ThreadLocalRandom;
import java.util.stream.Collectors;
@@ -105,7 +106,7 @@
builder.setDimension(validateDimension(node));
builder.setSimilarity(validateSimilarity(node));
builder.setQuantization(validateQuantization(node));
- builder.setTrainListFraction(validateTrainList(node));
+ validateTrainList(node).ifPresent(builder::setTrainListFraction);
builder.setEpsilon(validateEpsilon(node));
validateNumClusters(node).ifPresent(builder::setNumClusters);
builder.setCrossPollinationM(validateCrossPollinationM(node));
@@ -190,10 +191,12 @@
/**
* Training list size is specified only via {@code train_list_fraction} (with ANALYZE/cardinality at build time).
*/
- private static double validateTrainList(AdmObjectNode node) throws CompilationException {
+ private static OptionalDouble validateTrainList(AdmObjectNode node) throws CompilationException {
IAdmNode fn = node.get(TRAIN_LIST_FRACTION);
if (fn == null) {
- return VectorIndexParameters.DEFAULT_TRAIN_LIST_FRACTION;
+ // Left unset rather than defaulted: the size is derived from the cluster count when the user
+ // says nothing, and a defaulted value here would be indistinguishable from one they typed.
+ return OptionalDouble.empty();
}
double value = parseDoubleOrBigInt(fn,
"Invalid `train_list_fraction` parameter value. It must be in the range of (0,1]");
@@ -201,7 +204,7 @@
throw new CompilationException(ErrorCode.COMPILATION_VECTOR_INDEX_CREATION_FAILED,
"Invalid `train_list_fraction` parameter value. It must be in the range of (0,1]");
}
- return value;
+ return OptionalDouble.of(value);
}
private static double validateEpsilon(AdmObjectNode node) throws CompilationException {
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryVectorOperationsHelper.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryVectorOperationsHelper.java
index ab47484..fe0ad63 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryVectorOperationsHelper.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryVectorOperationsHelper.java
@@ -111,10 +111,6 @@
private RecordDescriptor recordDesc;
private static final float DEFAULT_CONFIDENCE_INTERVAL = 0.99f;
- /** Minimum train-list sample size for static-structure build; below this after clamp → full scan. */
- private static final int TRAIN_LIST_MIN_SAMPLE_SIZE = 10000;
- /** Maximum train-list sample size cap for static-structure build. */
- private static final int TRAIN_LIST_MAX_SAMPLE_SIZE = 1000000;
private IVTreeBinaryAccessorFactory vectorAccessorFactory;
protected SecondaryVectorOperationsHelper(Dataset dataset, Index index, MetadataProvider metadataProvider,
@@ -139,22 +135,6 @@
return vectorAccessorFactory;
}
- private static int clampTrainListSampleSize(int sampleSize, long datasetCardinality) {
- if (datasetCardinality > 0) {
- sampleSize = (int) Math.min(sampleSize, datasetCardinality);
- }
- sampleSize = Math.max(sampleSize, TRAIN_LIST_MIN_SAMPLE_SIZE);
- sampleSize = Math.min(sampleSize, TRAIN_LIST_MAX_SAMPLE_SIZE);
- if (datasetCardinality > 0) {
- sampleSize = (int) Math.min(sampleSize, datasetCardinality);
- }
- return sampleSize;
- }
-
- private static boolean useFullScanForTrainList(int sampleSize) {
- return sampleSize < TRAIN_LIST_MIN_SAMPLE_SIZE;
- }
-
@Override
public JobSpecification buildStaticStructureJobSpec() throws AlgebricksException {
IDataFormat format = metadataProvider.getDataFormat();
@@ -199,30 +179,30 @@
Index.SampleIndexDetails sampleDetails = (Index.SampleIndexDetails) sampleIndex.getIndexDetails();
long datasetCardinality = sampleDetails.getSourceCardinality();
- int sampleSize = 0;
if (datasetCardinality <= 0) {
throw new CompilationException(ErrorCode.COMPILATION_ERROR, sourceLoc,
- "train_list_fraction requires ANALYZE DATASET to be run first to obtain dataset cardinality.");
+ "Vector index requires a non-empty collection; load the data and run ANALYZE DATASET first.");
}
- sampleSize = (int) Math.max(1, datasetCardinality * trainListFraction);
-
- // Clamp sample size before deciding between full scan vs sampling.
- // Minimum TRAIN_LIST_MIN_SAMPLE_SIZE; maximum TRAIN_LIST_MAX_SAMPLE_SIZE.
- // Full scan if dataset cardinality is below TRAIN_LIST_MIN_SAMPLE_SIZE after clamp.
- sampleSize = clampTrainListSampleSize(sampleSize, datasetCardinality);
- boolean useFullScan = useFullScanForTrainList(sampleSize);
PartitioningProperties partitioningProperties = metadataProvider.getPartitioningProperties(dataset);
int numPartitions = partitioningProperties.getNumberOfPartitions();
+ // The sizing was fixed on the DDL path and persisted, so this reads it back rather than deriving it
+ // again: a rebuild reproduces the original build even if the collection has grown since. Everything
+ // here is per storage partition, which is the unit the rules are expressed in and the unit the sample
+ // scan wants. There is no global figure to divide down.
+ long perPartitionCardinality =
+ VectorIndexSizeInference.perPartitionCardinality(datasetCardinality, numPartitions);
+ long trainListSize = Math.max(1L, Math.round(trainListFraction * perPartitionCardinality));
+ boolean useFullScan = VectorIndexSizeInference.useFullScan(trainListSize, perPartitionCardinality);
+
IOperatorDescriptor sourceOp = DatasetUtil.createDummyKeyProviderOp(spec, dataset, metadataProvider);
IOperatorDescriptor targetOp;
if (useFullScan) {
targetOp = DatasetUtil.createPrimaryIndexScanOp(spec, metadataProvider, dataset, projectorFactory);
} else {
- int sampleCardinalityPerPartition = Math.max(1, sampleSize / numPartitions);
- targetOp = DatasetUtil.createSampleScanOp(spec, metadataProvider, dataset, sampleCardinalityPerPartition,
- seed, projectorFactory);
+ targetOp = DatasetUtil.createSampleScanOp(spec, metadataProvider, dataset,
+ (int) Math.min(trainListSize, Integer.MAX_VALUE), seed, projectorFactory);
}
spec.connect(new OneToOneConnectorDescriptor(spec), sourceOp, 0, targetOp, 0);
@@ -243,8 +223,10 @@
// k-means cannot be run with K = 0. An explicit num_clusters is already validated as >= 1 by
// VectorIndexDeclUtil.validateNumClusters, so the clamp only affects the computed default.
VectorIndexParameters vectorParameters = indexDetails.getVectorParameters();
- int defaultNumClusters = Math.max(1, (int) Math.sqrt((double) datasetCardinality / numPartitions));
- int K = vectorParameters.getNumClusters().orElse(defaultNumClusters);
+ // Present for every index created since the sizing moved to the DDL path. The fallback covers an
+ // index created before that, whose record never carried the cluster count.
+ int K = vectorParameters.getNumClusters()
+ .orElseGet(() -> VectorIndexSizeInference.defaultNumClusters(perPartitionCardinality));
// Distance metric from index DDL (WITH similarity "euclidean"|"cosine"|"cosine similarity"|etc.).
// For cosine, embeddings must be L2-normalized to unit length before insert; the engine does not normalize.
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/VectorIndexSizeInference.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/VectorIndexSizeInference.java
new file mode 100644
index 0000000..eeb95f5
--- /dev/null
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/VectorIndexSizeInference.java
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.asterix.metadata.utils;
+
+import java.util.OptionalDouble;
+
+import org.apache.asterix.common.exceptions.CompilationException;
+import org.apache.asterix.common.exceptions.ErrorCode;
+import org.apache.asterix.metadata.declared.MetadataProvider;
+import org.apache.asterix.metadata.entities.Dataset;
+import org.apache.asterix.metadata.entities.Index;
+import org.apache.asterix.om.vector.VectorIndexParameters;
+import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException;
+import org.apache.hyracks.api.exceptions.IWarningCollector;
+import org.apache.hyracks.api.exceptions.SourceLocation;
+import org.apache.hyracks.api.exceptions.Warning;
+import org.apache.hyracks.util.annotations.AiProvenance;
+
+/**
+ * Infers the two sizes a vector index build needs when the user does not state them: the leaf cluster count
+ * and the training list the k-means sees.
+ * <p>
+ * Everything here is expressed in <em>per storage partition</em> terms, because that is the unit a build
+ * actually works in, since each partition trains its own tree over its own rows. {@code D} throughout is the
+ * per-partition cardinality.
+ * <p>
+ * The two knobs are not independent. The training list is derived from the cluster count at
+ * {@link #TRAIN_ROWS_PER_CLUSTER} rows per cluster, which is the rule of thumb for having enough data to pick
+ * a representative centroid. That coupling is why the two cannot both be set by the user.
+ */
+@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_CLI, contributionKind = AiProvenance.ContributionKind.ASSISTED)
+public final class VectorIndexSizeInference {
+
+ /**
+ * Below this, {@code sqrt(D)} would give clusters too small to train on, so the small-data rule applies
+ * instead. At the boundary the two rules agree: {@code D = 10_000} gives 100 clusters either way.
+ */
+ private static final int SMALL_DATA_SQRT_THRESHOLD = 100;
+
+ /** Target rows per cluster in the small-data branch. */
+ private static final int SMALL_DATA_CLUSTER_ROWS = 100;
+
+ /** Training rows per cluster: the rule of thumb for picking representative centroids. */
+ public static final int TRAIN_ROWS_PER_CLUSTER = 50;
+
+ private VectorIndexSizeInference() {
+ }
+
+ /** The per-partition cardinality {@code D}, the unit every rule below is expressed in. */
+ public static long perPartitionCardinality(long sourceCardinality, int numPartitions) {
+ return Math.max(1L, sourceCardinality / Math.max(1, numPartitions));
+ }
+
+ /**
+ * The leaf cluster count when the user does not give one: {@code sqrt(D)}, or {@code ceil(D/100)} on small
+ * data where {@code sqrt(D)} would leave too few rows per cluster to train on.
+ */
+ public static int defaultNumClusters(long d) {
+ long k = isSmallData(d) ? (long) Math.ceil((double) d / SMALL_DATA_CLUSTER_ROWS) : (long) Math.sqrt((double) d);
+ return (int) Math.max(1L, Math.min(k, Integer.MAX_VALUE));
+ }
+
+ /** True when {@code sqrt(D)} is too small to be a useful cluster count, so the build reads every row. */
+ public static boolean isSmallData(long d) {
+ return Math.sqrt((double) d) < SMALL_DATA_SQRT_THRESHOLD;
+ }
+
+ /**
+ * The training list a given cluster count needs, bounded by the partition and by the backstop ceiling.
+ * <p>
+ * Small data trains on the whole partition rather than on {@code 50 * numClusters}. The build reads every
+ * row there anyway, because there is no sampling operator and nothing truncates the scan, so training on a
+ * subset would mean discarding rows already in hand to land exactly on a floor. The rule is a floor: the
+ * spec accepts any list at or above {@code 50 * numClusters}, and the whole partition clears it by more.
+ * Recording the whole partition is also what keeps the persisted fraction equal to what the build used.
+ */
+ public static long trainListSize(int numClusters, long d) {
+ if (isSmallData(d)) {
+ return d;
+ }
+ long wanted = (long) TRAIN_ROWS_PER_CLUSTER * numClusters;
+ return Math.max(1L, Math.min(wanted, d));
+ }
+
+ /**
+ * Whether the build should read every row instead of sampling: true exactly when the training list is the
+ * whole partition. An integer comparison rather than a test on the fraction, which would be a float
+ * landing on 1.0.
+ */
+ public static boolean useFullScan(long trainListSize, long d) {
+ return trainListSize >= d;
+ }
+
+ /**
+ * The training list an explicitly requested fraction asks for. Bounded only by the partition: an explicit
+ * fraction is honoured as written, so that what is recorded is what the user asked for. Case 3 of the
+ * model warns about a fraction that is too small; it puts no ceiling on one that is large.
+ */
+ public static long trainListSizeForFraction(double fraction, long d) {
+ return Math.max(1L, Math.min((long) Math.ceil(fraction * d), d));
+ }
+
+ /**
+ * The fraction to record for a training list of {@code size}. Small data reads every row, so it records
+ * 1.0: what is persisted is what the build used, not what motivated the cluster count.
+ */
+ public static double effectiveFraction(long trainListSize, long d) {
+ if (d <= 0L) {
+ return 1.0;
+ }
+ return Math.min(1.0, (double) trainListSize / d);
+ }
+
+ /**
+ * Infers the sizes a build will use and returns the parameters carrying them, so the index record states
+ * what was used rather than leaving it to be re-derived later from a cardinality that has moved. Called
+ * on the DDL path, before the record is written; the seed is fixed at the same point and for the same
+ * reason.
+ * <p>
+ * Both inputs are compile-time facts, so both diagnostics below are compile-time errors, which is where
+ * the model puts them: a missing ANALYZE and an empty collection.
+ */
+ public static VectorIndexParameters infer(MetadataProvider metadataProvider, Dataset dataset,
+ VectorIndexParameters params, IWarningCollector warningCollector, SourceLocation sourceLoc)
+ throws AlgebricksException {
+ Index sampleIndex = metadataProvider.findSampleIndex(dataset.getDatabaseName(), dataset.getDataverseName(),
+ dataset.getDatasetName());
+ if (sampleIndex == null || !(sampleIndex.getIndexDetails() instanceof Index.SampleIndexDetails)) {
+ throw new CompilationException(ErrorCode.COMPILATION_ERROR, sourceLoc,
+ "Run ANALYZE on the dataset before creating a vector index.");
+ }
+ long cardinality = ((Index.SampleIndexDetails) sampleIndex.getIndexDetails()).getSourceCardinality();
+ if (cardinality <= 0L) {
+ throw new CompilationException(ErrorCode.COMPILATION_ERROR, sourceLoc,
+ "Vector index requires a non-empty collection; load the data and run ANALYZE DATASET first.");
+ }
+ int numPartitions = metadataProvider.getPartitioningProperties(dataset).getNumberOfPartitions();
+ final long d = perPartitionCardinality(cardinality, numPartitions);
+ int recommendedClusters = defaultNumClusters(d);
+ int numClusters = params.getNumClusters().orElse(recommendedClusters);
+ if (params.getNumClusters().isPresent()) {
+ validateNumClusters(numClusters, recommendedClusters, d, warningCollector, sourceLoc);
+ }
+ OptionalDouble requested = params.getTrainListFractionOpt();
+ long size = requested.isPresent() ? trainListSizeForFraction(requested.getAsDouble(), d)
+ : trainListSize(numClusters, d);
+ if (requested.isPresent()) {
+ validateTrainList(requested.getAsDouble(), size, numClusters, d, warningCollector, sourceLoc);
+ }
+ return params.withInferredSizes(numClusters, effectiveFraction(size, d));
+ }
+
+ /**
+ * Case 2 of the model. More clusters than rows cannot be built, because the padding path would invent
+ * the shortfall as perturbed duplicates of real centroids, so that is an error. Otherwise a value far from
+ * what the collection suggests is only a warning: it is legal, and the user may know something the
+ * cardinality does not say.
+ * <p>
+ * The recommendation is whatever {@link #defaultNumClusters} would have chosen, not {@code sqrt(D)}
+ * unconditionally. On a small collection those differ, and recommending {@code sqrt(D)} there would warn
+ * about the value this same code picks when the user says nothing.
+ */
+ static void validateNumClusters(int numClusters, int recommended, long d, IWarningCollector warningCollector,
+ SourceLocation sourceLoc) throws CompilationException {
+ if (numClusters > d) {
+ throw new CompilationException(ErrorCode.COMPILATION_VECTOR_INDEX_CREATION_FAILED, sourceLoc,
+ "num_clusters " + numClusters + " exceeds the " + d
+ + " records per storage partition available to train on. " + recommended
+ + " is recommended.");
+ }
+ if ((numClusters < 0.9 * recommended || numClusters > 1.1 * recommended) && warningCollector.shouldWarn()) {
+ warningCollector.warn(Warning.of(sourceLoc, ErrorCode.VECTOR_INDEX_NUM_CLUSTERS_NOT_RECOMMENDED,
+ numClusters, recommended, d));
+ }
+ }
+
+ /**
+ * Case 3 of the model. A training list smaller than the cluster count cannot produce a centroid per
+ * cluster, so that is an error; below the rows-per-cluster rule of thumb it is a warning, since the
+ * centroids will be built from thin evidence but can still be built.
+ */
+ static void validateTrainList(double fraction, long size, int numClusters, long d,
+ IWarningCollector warningCollector, SourceLocation sourceLoc) throws CompilationException {
+ if (size < numClusters) {
+ throw new CompilationException(ErrorCode.COMPILATION_VECTOR_INDEX_CREATION_FAILED, sourceLoc,
+ "train_list_fraction " + fraction + " trains on " + size + " records per storage partition, "
+ + "fewer than the " + numClusters + " clusters to be built. Raise it to at least "
+ + recommendedFraction(numClusters, d) + ".");
+ }
+ long recommendedSize = (long) TRAIN_ROWS_PER_CLUSTER * numClusters;
+ if (size < recommendedSize && warningCollector.shouldWarn()) {
+ warningCollector.warn(Warning.of(sourceLoc, ErrorCode.VECTOR_INDEX_TRAIN_LIST_TOO_SMALL, fraction, size,
+ recommendedSize, numClusters, recommendedFraction(numClusters, d)));
+ }
+ }
+
+ /** The smallest fraction that reaches the rows-per-cluster rule of thumb, for use in a recommendation. */
+ private static double recommendedFraction(int numClusters, long d) {
+ return Math.min(1.0, (double) TRAIN_ROWS_PER_CLUSTER * numClusters / d);
+ }
+}
diff --git a/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/VectorIndexParametersTupleTranslatorTest.java b/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/VectorIndexParametersTupleTranslatorTest.java
index 78ffa7f..c376a15 100644
--- a/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/VectorIndexParametersTupleTranslatorTest.java
+++ b/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/entitytupletranslators/VectorIndexParametersTupleTranslatorTest.java
@@ -31,6 +31,7 @@
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.OptionalDouble;
import java.util.OptionalInt;
import org.apache.asterix.common.config.DatasetConfig.DatasetType;
@@ -71,7 +72,7 @@
* without extending {@code NAMES} fails it, and the failure message names the methods to update. Give the new
* parameter a non-default value in {@link #everyParameterRoundTrips} and the round trip is covered too.
*/
-@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.TEST_GENERATED)
+@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.ASSISTED)
public class VectorIndexParametersTupleTranslatorTest {
/** Every parameter set to a non-default value, so a dropped field cannot look like a correct round trip. */
@@ -112,7 +113,17 @@
VectorIndexParameters readBack = roundTrip(minimal);
- Assert.assertEquals(minimal, readBack);
+ // Not equal to the input: train_list_fraction was left unset, and unset is a DDL-time state meaning
+ // "derive this from the cardinality". Nothing persists it in that state, so a record that does not
+ // carry it reads back at the historical default instead, which is what keeps an index created before
+ // the parameter existed behaving as it did.
+ Assert.assertEquals(OptionalDouble.empty(), minimal.getTrainListFractionOpt());
+ Assert.assertEquals(OptionalDouble.of(VectorIndexParameters.DEFAULT_TRAIN_LIST_FRACTION),
+ readBack.getTrainListFractionOpt());
+ Assert.assertEquals(
+ minimal.withInferredSizes(readBack.getNumClusters().orElse(1),
+ VectorIndexParameters.DEFAULT_TRAIN_LIST_FRACTION).getTrainListFraction(),
+ readBack.getTrainListFraction(), 0.0);
Assert.assertEquals(OptionalInt.empty(), readBack.getNumClusters());
// 0 is a seed like any other, not an "unset" marker: it must survive the round trip rather than being
// treated as absent and redrawn.
diff --git a/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/utils/VectorIndexSizeInferenceTest.java b/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/utils/VectorIndexSizeInferenceTest.java
new file mode 100644
index 0000000..575ade9
--- /dev/null
+++ b/asterixdb/asterix-metadata/src/test/java/org/apache/asterix/metadata/utils/VectorIndexSizeInferenceTest.java
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.asterix.metadata.utils;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.asterix.common.exceptions.CompilationException;
+import org.apache.hyracks.api.exceptions.IWarningCollector;
+import org.apache.hyracks.api.exceptions.Warning;
+import org.apache.hyracks.util.annotations.AiProvenance;
+import org.junit.Test;
+
+/**
+ * The sizing rules, checked against the worked examples in the design they implement. Pure arithmetic, so it
+ * is verified here rather than through a cluster; the runtime test only has to show the values are persisted.
+ */
+@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_CLI, contributionKind = AiProvenance.ContributionKind.ASSISTED)
+public class VectorIndexSizeInferenceTest {
+
+ /** D = 900: 9 clusters. The design's small-data example. */
+ @Test
+ public void smallDataUsesHundredRowClusters() {
+ assertTrue(VectorIndexSizeInference.isSmallData(900));
+ assertEquals(9, VectorIndexSizeInference.defaultNumClusters(900));
+ }
+
+ /**
+ * Small data trains on the whole partition. 50*9 = 450 is the floor the design states, and reading every
+ * row clears it: the scan has already produced them and nothing truncates it.
+ */
+ @Test
+ public void smallDataTrainsOnEverythingAndFullScans() {
+ long size = VectorIndexSizeInference.trainListSize(9, 900);
+ assertEquals(900, size);
+ assertTrue(size >= 50L * 9);
+ assertTrue(VectorIndexSizeInference.useFullScan(size, 900));
+ assertEquals(1.0, VectorIndexSizeInference.effectiveFraction(size, 900), 0.0);
+ }
+
+ /** The three non-small worked examples: 100/50%, 200/25%, 300/16.7%. */
+ @Test
+ public void reproducesTheWorkedExamples() {
+ assertSizing(10_000, 100, 5_000, 0.5);
+ assertSizing(40_000, 200, 10_000, 0.25);
+ assertSizing(90_000, 300, 15_000, 1.0 / 6);
+ }
+
+ /** The two branches agree at the boundary, so the rule has no step in it. */
+ @Test
+ public void branchesMeetAtTheThreshold() {
+ assertFalse(VectorIndexSizeInference.isSmallData(10_000));
+ assertEquals(100, VectorIndexSizeInference.defaultNumClusters(10_000));
+ assertEquals(100, (int) Math.ceil(10_000 / 100.0));
+ }
+
+ /** The training list grows as sqrt(D), so a bigger collection samples a smaller share, not a larger one. */
+ @Test
+ public void trainingListGrowsSublinearly() {
+ assertSizing(1_000_000, 1_000, 50_000, 0.05);
+ double small =
+ VectorIndexSizeInference.effectiveFraction(VectorIndexSizeInference.trainListSize(100, 10_000), 10_000);
+ double large = VectorIndexSizeInference
+ .effectiveFraction(VectorIndexSizeInference.trainListSize(1_000, 1_000_000), 1_000_000);
+ assertTrue(large < small);
+ }
+
+ /** An explicit fraction is honoured as written; nothing caps it back. */
+ @Test
+ public void explicitFractionIsHonoured() {
+ assertEquals(900, VectorIndexSizeInference.trainListSizeForFraction(1.0, 900));
+ assertEquals(90, VectorIndexSizeInference.trainListSizeForFraction(0.1, 900));
+ assertEquals(500_000, VectorIndexSizeInference.trainListSizeForFraction(0.5, 1_000_000));
+ }
+
+ /** Fewer rows than partitions: D floors at 1 rather than 0, which k-means cannot run on. */
+ @Test
+ public void fewerRowsThanPartitions() {
+ assertEquals(1, VectorIndexSizeInference.perPartitionCardinality(3, 8));
+ assertEquals(1, VectorIndexSizeInference.defaultNumClusters(1));
+ assertTrue(VectorIndexSizeInference.useFullScan(VectorIndexSizeInference.trainListSize(1, 1), 1));
+ }
+
+ /** The fraction is scale-invariant: derived per partition, it is the same number globally. */
+ @Test
+ public void fractionIsScaleInvariant() {
+ int partitions = 8;
+ long global = 8_000_000L;
+ long d = VectorIndexSizeInference.perPartitionCardinality(global, partitions);
+ int k = VectorIndexSizeInference.defaultNumClusters(d);
+ long perPartition = VectorIndexSizeInference.trainListSize(k, d);
+ assertEquals(VectorIndexSizeInference.effectiveFraction(perPartition, d),
+ (double) (perPartition * partitions) / global, 1e-12);
+ }
+
+ private static void assertSizing(long d, int expectedClusters, long expectedSize, double expectedFraction) {
+ int k = VectorIndexSizeInference.defaultNumClusters(d);
+ assertEquals("clusters for D=" + d, expectedClusters, k);
+ long size = VectorIndexSizeInference.trainListSize(k, d);
+ assertEquals("training list for D=" + d, expectedSize, size);
+ assertEquals("fraction for D=" + d, expectedFraction, VectorIndexSizeInference.effectiveFraction(size, d),
+ 1e-9);
+ assertFalse("D=" + d + " should sample, not full scan", VectorIndexSizeInference.useFullScan(size, d));
+ }
+
+ /** Case 2: more clusters than rows cannot be built, so it is an error rather than a warning. */
+ @Test
+ public void moreClustersThanRowsIsAnError() throws Exception {
+ CollectingWarnings warnings = new CollectingWarnings();
+ try {
+ VectorIndexSizeInference.validateNumClusters(5000, VectorIndexSizeInference.defaultNumClusters(30), 30,
+ warnings, null);
+ fail("expected a compilation error");
+ } catch (CompilationException e) {
+ assertTrue(e.getMessage(), e.getMessage().contains("records per storage partition available to train on"));
+ }
+ assertTrue("an error must not also warn", warnings.messages.isEmpty());
+ }
+
+ /** Case 2: a value far from the recommendation is legal, so it warns and the build proceeds. */
+ @Test
+ public void clustersFarFromRecommendedWarns() throws Exception {
+ CollectingWarnings warnings = new CollectingWarnings();
+ VectorIndexSizeInference.validateNumClusters(3, VectorIndexSizeInference.defaultNumClusters(30), 30, warnings,
+ null);
+ assertEquals(1, warnings.messages.size());
+ }
+
+ /** Case 2: inside the +/-10% window nothing is said. */
+ @Test
+ public void clustersNearRecommendedAreSilent() throws Exception {
+ CollectingWarnings warnings = new CollectingWarnings();
+ int recommended = VectorIndexSizeInference.defaultNumClusters(1_000_000);
+ VectorIndexSizeInference.validateNumClusters(recommended, recommended, 1_000_000, warnings, null);
+ VectorIndexSizeInference.validateNumClusters((int) (recommended * 1.05), recommended, 1_000_000, warnings,
+ null);
+ assertTrue(warnings.messages.toString(), warnings.messages.isEmpty());
+ }
+
+ /** Case 3: fewer training rows than clusters cannot place a centroid per cluster, so it is an error. */
+ @Test
+ public void trainListSmallerThanClusterCountIsAnError() throws Exception {
+ CollectingWarnings warnings = new CollectingWarnings();
+ try {
+ VectorIndexSizeInference.validateTrainList(0.01, 1, 8, 900, warnings, null);
+ fail("expected a compilation error");
+ } catch (CompilationException e) {
+ assertTrue(e.getMessage(), e.getMessage().contains("fewer than the 8 clusters to be built"));
+ }
+ }
+
+ /** Case 3: above the cluster count but below the rows-per-cluster floor is a warning, not an error. */
+ @Test
+ public void trainListBelowTheFloorWarns() throws Exception {
+ CollectingWarnings warnings = new CollectingWarnings();
+ VectorIndexSizeInference.validateTrainList(0.1, 90, 9, 900, warnings, null);
+ assertEquals(1, warnings.messages.size());
+ }
+
+ /** Case 3: at or above the floor nothing is said. */
+ @Test
+ public void trainListAtTheFloorIsSilent() throws Exception {
+ CollectingWarnings warnings = new CollectingWarnings();
+ VectorIndexSizeInference.validateTrainList(0.5, 450, 9, 900, warnings, null);
+ assertTrue(warnings.messages.toString(), warnings.messages.isEmpty());
+ }
+
+ private static final class CollectingWarnings implements IWarningCollector {
+ private final List<Warning> messages = new ArrayList<>();
+
+ @Override
+ public void warn(Warning warning) {
+ messages.add(warning);
+ }
+
+ @Override
+ public boolean shouldWarn() {
+ return true;
+ }
+
+ @Override
+ public long getTotalWarningsCount() {
+ return messages.size();
+ }
+ }
+}
diff --git a/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/vector/VectorIndexParameters.java b/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/vector/VectorIndexParameters.java
index 2e5032d..fc848e4 100644
--- a/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/vector/VectorIndexParameters.java
+++ b/asterixdb/asterix-om/src/main/java/org/apache/asterix/om/vector/VectorIndexParameters.java
@@ -21,6 +21,7 @@
import java.io.Serializable;
import java.util.List;
import java.util.Objects;
+import java.util.OptionalDouble;
import java.util.OptionalInt;
import java.util.concurrent.ThreadLocalRandom;
@@ -69,7 +70,7 @@
* reloaded — the defect this class was extracted to prevent. {@code VectorIndexParametersTupleTranslatorTest}
* guards the count: add a field without extending {@code NAMES} and it fails.
*/
-@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.GENERATED)
+@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.ASSISTED)
public final class VectorIndexParameters implements Serializable {
private static final long serialVersionUID = 1L;
@@ -87,6 +88,11 @@
public static final String SEED = "seed";
public static final VectorQuantization DEFAULT_QUANTIZATION = VectorQuantization.SQ8;
+ /**
+ * The fraction an index record reads back as when it does not carry one. Not a default anyone applies
+ * any more: an index created since the sizing moved to the DDL path always records the fraction its
+ * build used, so this only stands in for a record written before that.
+ */
public static final double DEFAULT_TRAIN_LIST_FRACTION = 0.1;
public static final double DEFAULT_EPSILON = 0.25;
public static final int DEFAULT_CROSS_POLLINATION_M = 1;
@@ -104,7 +110,12 @@
private final int dimension;
private final VectorSimilarityMetric similarity;
private final VectorQuantization quantization;
- private final double trainListFraction;
+ /**
+ * {@code null} when unset: the training-list size is then derived from the cluster count. Nullable for
+ * the same reason {@link #numClusters} is. A primitive cannot distinguish "the user asked for 0.1"
+ * from "the user said nothing", and those two now size the training list differently.
+ */
+ private final Double trainListFraction;
private final double epsilon;
/** {@code null} when unset: the builder then derives it from the dataset cardinality at build time. */
private final Integer numClusters;
@@ -112,6 +123,36 @@
private final double rngFactor;
private final long seed;
+ /**
+ * Returns a copy with {@code num_clusters} and {@code train_list_fraction} fixed to the values the build
+ * will actually use.
+ * <p>
+ * Both are derived from the per-partition cardinality when the user does not supply them, so without this
+ * the record states the fraction the user asked for and says nothing at all about the cluster count
+ * ({@code num_clusters} is written only when set). What a build used would then be unrecoverable, since
+ * re-deriving it needs the cardinality as it stood at creation. This mirrors how the seed is fixed at DDL
+ * time and persisted so the index records what its own build used.
+ */
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_CLI, contributionKind = AiProvenance.ContributionKind.ASSISTED)
+ public VectorIndexParameters withInferredSizes(int effectiveNumClusters, double effectiveTrainListFraction) {
+ Builder b = builder();
+ b.setDimension(dimension);
+ b.setSimilarity(similarity);
+ b.setQuantization(quantization);
+ b.setTrainListFraction(effectiveTrainListFraction);
+ b.setEpsilon(epsilon);
+ b.setNumClusters(effectiveNumClusters);
+ b.setCrossPollinationM(crossPollinationM);
+ b.setRngFactor(rngFactor);
+ b.setSeed(seed);
+ try {
+ return b.build();
+ } catch (AsterixException e) {
+ // Unreachable: every mandatory parameter is copied from an instance that already built.
+ throw new IllegalStateException(e);
+ }
+ }
+
private VectorIndexParameters(Builder builder) {
this.dimension = builder.dimension;
this.similarity = builder.similarity;
@@ -165,10 +206,22 @@
return true;
}
+ /**
+ * The effective fraction. Valid only once sizing has been derived, which the DDL path does before the
+ * index record is written; {@link #getTrainListFractionOpt()} is the accessor for the state before that.
+ */
public double getTrainListFraction() {
+ if (trainListFraction == null) {
+ throw new IllegalStateException(TRAIN_LIST_FRACTION + " has not been derived yet");
+ }
return trainListFraction;
}
+ /** Empty when the user did not specify it; the DDL path then derives it from the cluster count. */
+ public OptionalDouble getTrainListFractionOpt() {
+ return trainListFraction == null ? OptionalDouble.empty() : OptionalDouble.of(trainListFraction);
+ }
+
public double getEpsilon() {
return epsilon;
}
@@ -224,7 +277,9 @@
writer.writeInt(DIMENSION, dimension);
writer.writeString(SIMILARITY, similarity.canonical());
writer.writeString(QUANTIZATION, quantization.label());
- writer.writeDouble(TRAIN_LIST_FRACTION, trainListFraction);
+ if (trainListFraction != null) {
+ writer.writeDouble(TRAIN_LIST_FRACTION, trainListFraction);
+ }
writer.writeDouble(EPSILON, epsilon);
if (numClusters != null) {
writer.writeInt(NUM_CLUSTERS, numClusters);
@@ -274,9 +329,10 @@
builder.setQuantization(quantization);
}
Double trainListFraction = reader.readDouble(TRAIN_LIST_FRACTION);
- if (trainListFraction != null) {
- builder.setTrainListFraction(trainListFraction);
- }
+ // Always concrete when read back. "Unset" is a DDL-time state meaning "derive this from the
+ // cardinality"; an index already in the catalog has been sized, so a record that predates the
+ // parameter falls back to the historical default rather than reading back as unset.
+ builder.setTrainListFraction(trainListFraction != null ? trainListFraction : DEFAULT_TRAIN_LIST_FRACTION);
Double epsilon = reader.readDouble(EPSILON);
if (epsilon != null) {
builder.setEpsilon(epsilon);
@@ -310,7 +366,7 @@
}
return dimension == other.dimension && Objects.equals(similarity, other.similarity)
&& Objects.equals(quantization, other.quantization)
- && Double.compare(trainListFraction, other.trainListFraction) == 0
+ && Objects.equals(trainListFraction, other.trainListFraction)
&& Double.compare(epsilon, other.epsilon) == 0 && Objects.equals(numClusters, other.numClusters)
&& crossPollinationM == other.crossPollinationM && Double.compare(rngFactor, other.rngFactor) == 0
&& seed == other.seed;
@@ -350,7 +406,7 @@
private int dimension = -1;
private VectorSimilarityMetric similarity;
private VectorQuantization quantization = DEFAULT_QUANTIZATION;
- private double trainListFraction = DEFAULT_TRAIN_LIST_FRACTION;
+ private Double trainListFraction;
private double epsilon = DEFAULT_EPSILON;
private Integer numClusters;
private int crossPollinationM = DEFAULT_CROSS_POLLINATION_M;
diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/QuantizedIndexCreateOperatorDescriptor.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/QuantizedIndexCreateOperatorDescriptor.java
index bcabada..03a2b52 100644
--- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/QuantizedIndexCreateOperatorDescriptor.java
+++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/QuantizedIndexCreateOperatorDescriptor.java
@@ -29,7 +29,6 @@
import org.apache.hyracks.api.dataflow.value.RecordDescriptor;
import org.apache.hyracks.api.exceptions.ErrorCode;
import org.apache.hyracks.api.exceptions.HyracksDataException;
-import org.apache.hyracks.util.annotations.AiProvenance;
import org.apache.hyracks.api.job.IOperatorDescriptorRegistry;
import org.apache.hyracks.data.std.primitive.ByteArrayPointable;
import org.apache.hyracks.dataflow.common.comm.io.FrameTupleAccessor;
@@ -39,6 +38,7 @@
import org.apache.hyracks.storage.am.common.api.IIndexBuilder;
import org.apache.hyracks.storage.am.common.api.IIndexBuilderFactory;
import org.apache.hyracks.storage.am.vector.api.VTreeQuantizationParams;
+import org.apache.hyracks.util.annotations.AiProvenance;
/**
* Operator descriptor for creating a quantized vector index.
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21652?usp=email
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I5d90fdd079f0931edaeb46a26ac2c717e07a043c
Gerrit-Change-Number: 21652
Gerrit-PatchSet: 1
Gerrit-Owner: Hongyu Shi <le...@gmail.com>