You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@asterixdb.apache.org by al...@apache.org on 2026/08/09 19:52:21 UTC
(asterixdb) branch master updated: [ASTERIXDB-3760][ASTERIXDB-3771][ASTERIXDB-3754] Vector index misc fixes
This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
The following commit(s) were added to refs/heads/master by this push:
new db88f805da [ASTERIXDB-3760][ASTERIXDB-3771][ASTERIXDB-3754] Vector index misc fixes
db88f805da is described below
commit db88f805da229bf7a41a6310485dce7fb9180543
Author: Ali Alsuliman <al...@gmail.com>
AuthorDate: Sat Aug 8 12:22:43 2026 -0700
[ASTERIXDB-3760][ASTERIXDB-3771][ASTERIXDB-3754] Vector index misc fixes
- user model changes: yes
- storage format changes: no
- interface changes: yes
Address the "likely bugs" findings from the review of the VTree vector
index stack (21099, 21100, 21159, 21287).
Storage - VTree (21099):
- VTree.handleMetadataPageOverflow: capture the split page's successor
before splitting and splice left -> new -> original. Splitting a
directory page that was not last in its chain terminated the right
half, orphaning every page after it (their data pages became
unreachable to the insert walk and to tryPhysicalDelete).
VTreeMetadataFrame.split now also preserves the frame's level instead
of hardcoding 0, and documents that re-linking is the caller's job.
- VTreeBulkLoader.end: allocate the static-structure pages with
takeBlock(), not a takePage() loop - the loop can hand back
non-contiguous ids and break the (base + i) destination math.
- VTreeSearchCursor: on every empty-cluster exit, release the abandoned
cluster's page and clear dataFrame/frameTuple
(markCurrentClusterEmpty). Leaving them set let the next hasNext()
read getNextPage() off the stale frame and silently resume the
abandoned cluster's chain, which is reachable because the LSM layer
re-points the cursor mid-cluster.
- VTree.handleDataPageOverflow: enforce the "directory page is empty"
precondition it relies on; it does not link the new page into the
data-page chain, so on a populated directory the page would be
invisible to search.
- VTreeStaticStructureBuilder.end: run the leaf resolve/upload pass
when the page it flushes is a leaf page. Previously that page was
published local-only and never uploaded in cloud deployments
(single-level structures, and end() on a partially-built one). The
pass is now idempotent so it cannot re-upload already-resolved
leaves.
Storage - LSM VTree (21100):
- LSMVTreeFileManager.validateStaticStructureFile: propagate I/O errors
instead of returning false. cleanupAndGetValidFiles reads false as
"no static structure" and deletes every data component, so a
transient failure (cloud hiccup, EMFILE, slow mount) destroyed the
whole index.
- collectVTreeFiles: list through IIOManager rather than java.io.File,
which under-reports on cloud deployments.
- LSMVTreeSearchCursor.isDeleted: restore the instanceof guard the
comment above it describes; a rebuilt ArrayTupleReference threw
ClassCastException mid-merge.
- Both cursors: validate in doOpen that the comparators cover
<distance, PK...> (LSMVTreeUtils.validateKeyComparators). A short
array silently shortened the key - merge order collapsing to
distance-only, and samePrimaryKey() becoming universally true.
- LSMVTreeTopKSearchCursor: assert the index is quantized and that a
quantizer and quantized query vector were supplied. Cursor selection
hangs off the unrelated USE_TOPK_SEARCH flag, so a non-quantized
index read field 3 as the quantized embedding and mis-located the
PKs.
- LSMVTreeSearchCursor: count K progress in doNext(), not doGetTuple().
getTuple() is not contractually once per tuple, so a second read
inflated progress and triggered early termination, reducing recall.
- File references: carry the shared static structure in
LSMVTreeComponentFileReferences' own slot instead of the bloom-filter
slot, on both the merge and the recovery paths.
- NprobeClusterSelectionStrategy: warn when epsilon <= 0 disables
level-wise probing entirely (the search silently degrades to a greedy
DFS from one cluster), and name the nprobe input for what it holds -
the epsilon-filtered candidate list, not the index's leaf clusters.
- VTreeSearchOperatorNodePushable.resetSearchPredicate: reset every
predicate slot on every input row. The predicate is reused, so "0
means default" actually meant "0 means the previous row's value".
Training (21159):
- SecondaryVectorOperationsHelper: clamp the computed num_clusters
default to at least 1 - sqrt(cardinality / partitions) is 0 for a
small dataset on a multi-partition cluster, and k-means cannot run
with K = 0.
- Seed the train-list sample from compiler.vector.trainseed as well, so
a sampled build (train_list_fraction < 1.0) is reproducible rather
than seeded off the wall clock.
DDL / query (21287):
- PushFilterIntoVectorSearchRule: match INCLUDE fields on the full path
instead of the leaf name, require the field access to be rooted at
this search's own record variable, and resolve nested
field-access-by-index against the nested type rather than the
top-level one. Any of the three could push a predicate against the
wrong column.
- VTreeSearchOperatorNodePushable: read integer query parameters by ADM
type tag. The fixed 4-byte read yielded 0 for an AInt64 k (LIMIT with
a BIGINT constant) after the compile-time narrowing was removed.
- VectorIndexAccessMethod: settle the index-only preconditions before
the job-gen params are serialized into the index-search arguments.
Clearing indexOnly afterwards was a no-op on the emitted plan while
the runtime still emitted [pk..., dist].
- QueryTranslator: reject a meta-sourced vector field (source
indicators are not persisted for a vector index - they are hardcoded
to RECORD_INDICATOR on read, so the index changed behaviour across a
restart), and reject a vector index on a dataset with a filter field,
which the index does not apply.
- MetadataProvider: stop appending the unused filter column to the
vector index's field permutation.
- SQLPP.jj: reject WITH on non-VTREE indexes and ENFORCED / CAST(...)
on VTREE ones, instead of parsing and silently discarding them.
- IntroduceTopKAccessMethodRule.chooseVectorIndex: bail out when the
query has no distance metric rather than taking the first vector
index found, which could answer a cosine query from a euclidean
index.
Ext-ref: MB-72975
Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
Change-Id: I5141f71c2cf36aa28f487d050709161cedee1460
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21500
Tested-by: Jenkins <je...@fulliautomatix.ics.uci.edu>
Reviewed-by: Murtadha Hubail <mh...@apache.org>
Reviewed-by: Ali Alsuliman <al...@gmail.com>
---
.../rules/PushFilterIntoVectorSearchRule.java | 223 +++++++++++++--------
.../rules/am/IntroduceTopKAccessMethodRule.java | 11 +-
.../rules/am/VectorIndexAccessMethod.java | 200 +++++++++---------
.../asterix/app/translator/QueryTranslator.java | 19 ++
.../asterix-lang-sqlpp/src/main/javacc/SQLPP.jj | 20 +-
.../metadata/declared/MetadataProvider.java | 23 +--
.../utils/SecondaryVectorOperationsHelper.java | 58 ++++--
.../am/lsm/common/impls/AbstractLSMIndex.java | 7 +-
.../dataflow/VTreeSearchOperatorNodePushable.java | 106 ++++++----
.../storage/am/lsm/vector/impls/LSMVTree.java | 31 +--
.../am/lsm/vector/impls/LSMVTreeFileManager.java | 59 +++---
.../am/lsm/vector/impls/LSMVTreeSearchCursor.java | 36 +++-
.../lsm/vector/impls/LSMVTreeTopKSearchCursor.java | 23 +++
.../impls/NprobeClusterSelectionStrategy.java | 25 ++-
.../storage/am/lsm/vector/utils/LSMVTreeUtils.java | 28 +++
.../am/vector/frames/VTreeMetadataFrame.java | 13 +-
.../hyracks/storage/am/vector/impls/VTree.java | 56 ++++--
.../storage/am/vector/impls/VTreeBulkLoader.java | 10 +-
.../storage/am/vector/impls/VTreeSearchCursor.java | 43 ++--
.../vector/impls/VTreeStaticStructureBuilder.java | 16 ++
20 files changed, 648 insertions(+), 359 deletions(-)
diff --git a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushFilterIntoVectorSearchRule.java b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushFilterIntoVectorSearchRule.java
index 67aaef6cbf..1c41b6fed0 100644
--- a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushFilterIntoVectorSearchRule.java
+++ b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushFilterIntoVectorSearchRule.java
@@ -32,6 +32,7 @@ import org.apache.asterix.metadata.entities.Index;
import org.apache.asterix.object.base.AdmObjectNode;
import org.apache.asterix.om.functions.BuiltinFunctions;
import org.apache.asterix.om.types.ARecordType;
+import org.apache.asterix.om.types.ATypeTag;
import org.apache.asterix.om.types.BuiltinType;
import org.apache.asterix.om.types.IAType;
import org.apache.asterix.optimizer.rules.am.AccessMethodJobGenParams;
@@ -55,6 +56,7 @@ import org.apache.hyracks.algebricks.core.algebra.operators.logical.UnnestMapOpe
import org.apache.hyracks.algebricks.core.algebra.util.OperatorPropertiesUtil;
import org.apache.hyracks.algebricks.core.rewriter.base.IAlgebraicRewriteRule;
import org.apache.hyracks.algebricks.rewriter.rules.InlineVariablesRule;
+import org.apache.hyracks.storage.am.vector.utils.VTreeDataTupleAccessor;
import org.apache.hyracks.util.annotations.AiProvenance;
/**
@@ -156,31 +158,34 @@ public class PushFilterIntoVectorSearchRule implements IAlgebraicRewriteRule {
}
}
- Set<String> filterFieldNames = new HashSet<>();
- extractFieldNames(conditionRef.getValue(), searchInfo.recordType(), filterFieldNames);
+ // Collect the FULL field paths the filter reads, resolved against the record variable(s) this vector
+ // search feeds. Matching on full paths (not leaf names) is what keeps `WHERE m.b.year > 2000` from
+ // being pushed against an `INCLUDE (a.year)` column, and restricting the base to this subtree's
+ // record variable(s) is what keeps a second record (from a join or a LET) from being redirected into
+ // the index's INCLUDE field.
+ Set<List<String>> filterFieldPaths = new HashSet<>();
+ extractFieldPaths(conditionRef.getValue(), searchInfo, filterFieldPaths);
- if (filterFieldNames.isEmpty()) {
+ if (filterFieldPaths.isEmpty()) {
return false;
}
// Bail out if the filter references any field that is not in the index's INCLUDE list.
- Map<String, Integer> includeFieldIndex = buildIncludeFieldIndex(searchInfo.includeFieldNames());
- // Leaf-name-keyed matching (here and in the variable-creation loop below) is ambiguous if two
- // INCLUDE fields share a leaf name -- only possible with nested paths, e.g. INCLUDE(a.x, b.x).
- // A collapsed map would silently bind the filter to whichever field was inserted last. Bail
- // rather than risk pushing a predicate that resolves to the wrong physical field.
+ Map<List<String>, Integer> includeFieldIndex = buildIncludeFieldIndex(searchInfo.includeFieldNames());
+ // Two identical INCLUDE paths would collapse the map and bind the filter to whichever was inserted
+ // last; bail rather than risk pushing a predicate that resolves to the wrong physical field.
if (includeFieldIndex.size() < searchInfo.includeFieldNames().size()) {
return false;
}
- for (String fieldName : filterFieldNames) {
- if (!includeFieldIndex.containsKey(fieldName)) {
+ for (List<String> fieldPath : filterFieldPaths) {
+ if (!includeFieldIndex.containsKey(fieldPath)) {
return false;
}
}
// Create fresh logical variables for INCLUDE fields referenced by the filter. These variables
// are produced by VECTOR_INDEX_UNNEST solely for filter evaluation.
- Map<String, LogicalVariable> fieldToNewVar = new HashMap<>();
+ Map<List<String>, LogicalVariable> fieldToNewVar = new HashMap<>();
Map<LogicalVariable, Integer> filterVarToFieldIndex = new HashMap<>();
Map<LogicalVariable, IAType> filterVarToType = new HashMap<>();
@@ -188,15 +193,14 @@ public class PushFilterIntoVectorSearchRule implements IAlgebraicRewriteRule {
// Non-quantized: [distance, centroidId, pk_0..pk_{N-1}, include_fields...]
// Quantized: [distance, centroidId, qDist, qEmbed, pk_0..pk_{N-1}, include_fields...]
// INCLUDE fields start after the secondary keys and ALL primary key columns.
- int numSecondaryKeys = searchInfo.isQuantized() ? 4 : 2;
+ int numSecondaryKeys = searchInfo.isQuantized() ? VTreeDataTupleAccessor.Q_NUM_SECONDARY_FIELDS
+ : VTreeDataTupleAccessor.NQ_NUM_SECONDARY_FIELDS;
int includeFieldPhysicalIndex = numSecondaryKeys + searchInfo.numPrimaryKeys();
for (List<String> fieldPath : searchInfo.includeFieldNames()) {
- String fieldName = fieldPath.get(fieldPath.size() - 1);
-
- if (filterFieldNames.contains(fieldName)) {
+ if (filterFieldPaths.contains(fieldPath)) {
LogicalVariable newVar = context.newVar();
- fieldToNewVar.put(fieldName, newVar);
+ fieldToNewVar.put(fieldPath, newVar);
filterVarToFieldIndex.put(newVar, includeFieldPhysicalIndex);
// For open-schema fields not in the type declaration, getSubFieldType returns null;
@@ -212,8 +216,7 @@ public class PushFilterIntoVectorSearchRule implements IAlgebraicRewriteRule {
}
// Rewrite field-access expressions in the condition to reference the new INCLUDE variables.
- ILogicalExpression rewrittenCondition =
- rewriteFieldAccess(conditionRef.getValue(), fieldToNewVar, searchInfo.recordType());
+ ILogicalExpression rewrittenCondition = rewriteFieldAccess(conditionRef.getValue(), fieldToNewVar, searchInfo);
// Completeness guard: the pushed condition must reference ONLY the freshly-created INCLUDE
// variables. The earlier checks act on field names that extractFieldNames recognizes; an
@@ -232,7 +235,7 @@ public class PushFilterIntoVectorSearchRule implements IAlgebraicRewriteRule {
searchInfo.vectorUnnest().getAnnotations().put(VECTOR_FILTER_VAR_TYPES, filterVarToType);
// Register filter variables as produced by the unnest-map so sanity checks recognize them.
- for (Map.Entry<String, LogicalVariable> entry : fieldToNewVar.entrySet()) {
+ for (Map.Entry<List<String>, LogicalVariable> entry : fieldToNewVar.entrySet()) {
LogicalVariable var = entry.getValue();
IAType type = filterVarToType.get(var);
searchInfo.vectorUnnest().getVariables().add(var);
@@ -254,9 +257,13 @@ public class PushFilterIntoVectorSearchRule implements IAlgebraicRewriteRule {
/**
* Information about a vector index search found in the plan.
+ *
+ * @param recordVars the variables produced between the SELECT and the vector search — i.e. the record
+ * (and PK) variables of THIS search's primary-index lookup. A field access must be
+ * rooted at one of these to be a candidate for pushdown.
*/
private record VectorSearchInfo(UnnestMapOperator vectorUnnest, List<List<String>> includeFieldNames,
- ARecordType recordType, boolean isQuantized, int numPrimaryKeys) {
+ ARecordType recordType, boolean isQuantized, int numPrimaryKeys, Set<LogicalVariable> recordVars) {
}
/**
@@ -268,14 +275,16 @@ public class PushFilterIntoVectorSearchRule implements IAlgebraicRewriteRule {
while (current.getOperatorTag() == LogicalOperatorTag.ASSIGN) {
current = current.getInputs().get(0).getValue();
}
- return searchForVectorUnnest(current, context);
+ return searchForVectorUnnest(current, context, new HashSet<>());
}
/**
- * Recursively searches for a VECTOR_INDEX_UNNEST under the given operator.
+ * Recursively searches for a VECTOR_INDEX_UNNEST under the given operator, accumulating on the way down
+ * the variables produced by the non-vector unnest-maps it passes through — the primary-index lookup that
+ * materializes the dataset record the filter reads.
*/
- private VectorSearchInfo searchForVectorUnnest(ILogicalOperator op, IOptimizationContext context)
- throws AlgebricksException {
+ private VectorSearchInfo searchForVectorUnnest(ILogicalOperator op, IOptimizationContext context,
+ Set<LogicalVariable> recordVars) throws AlgebricksException {
if (op.getOperatorTag() == LogicalOperatorTag.UNNEST_MAP) {
UnnestMapOperator unnest = (UnnestMapOperator) op;
@@ -292,14 +301,15 @@ public class PushFilterIntoVectorSearchRule implements IAlgebraicRewriteRule {
if (unnest.getSelectCondition() != null) {
return null;
}
- return buildSearchInfo(unnest, params, context);
+ return buildSearchInfo(unnest, params, context, recordVars);
}
}
}
+ recordVars.addAll(unnest.getVariables());
}
for (Mutable<ILogicalOperator> inputRef : op.getInputs()) {
- VectorSearchInfo result = searchForVectorUnnest(inputRef.getValue(), context);
+ VectorSearchInfo result = searchForVectorUnnest(inputRef.getValue(), context, recordVars);
if (result != null) {
return result;
}
@@ -313,7 +323,7 @@ public class PushFilterIntoVectorSearchRule implements IAlgebraicRewriteRule {
*/
@AiProvenance(agent = AiProvenance.Agent.CLAUDE_FABLE_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.ASSISTED)
private VectorSearchInfo buildSearchInfo(UnnestMapOperator unnest, AccessMethodJobGenParams params,
- IOptimizationContext context) throws AlgebricksException {
+ IOptimizationContext context, Set<LogicalVariable> recordVars) throws AlgebricksException {
MetadataProvider mp = (MetadataProvider) context.getMetadataProvider();
@@ -338,7 +348,7 @@ public class PushFilterIntoVectorSearchRule implements IAlgebraicRewriteRule {
dataset.getItemTypeDataverseName(), dataset.getItemTypeName());
return new VectorSearchInfo(unnest, details.getIncludeFieldNames(), recordType, quantization != null,
- dataset.getPrimaryKeys().size());
+ dataset.getPrimaryKeys().size(), recordVars);
}
/**
@@ -361,90 +371,143 @@ public class PushFilterIntoVectorSearchRule implements IAlgebraicRewriteRule {
}
/**
- * Builds a map from field name to its index in the INCLUDE list.
+ * Builds a map from an INCLUDE field's full path to its index in the INCLUDE list.
+ * <p>
+ * Keyed by the whole path, not the leaf name: {@code INCLUDE (a.year)} must not match a filter on
+ * {@code b.year}. {@link List} equality gives exactly path equality.
*/
- private Map<String, Integer> buildIncludeFieldIndex(List<List<String>> includeFieldNames) {
- Map<String, Integer> result = new HashMap<>();
+ private Map<List<String>, Integer> buildIncludeFieldIndex(List<List<String>> includeFieldNames) {
+ Map<List<String>, Integer> result = new HashMap<>();
for (int i = 0; i < includeFieldNames.size(); i++) {
- List<String> path = includeFieldNames.get(i);
- String fieldName = path.get(path.size() - 1);
- result.put(fieldName, i);
+ result.put(includeFieldNames.get(i), i);
}
return result;
}
/**
- * Extracts field names from field-access expressions in the filter.
+ * Collects the full field paths the filter reads from this vector search's record variable(s).
+ * <p>
+ * Only accesses rooted at one of {@code searchInfo.recordVars()} are collected. An access on any other
+ * record is deliberately ignored here, so it survives the rewrite and trips the completeness guard in
+ * {@link #rewritePre} — the rule then leaves the SELECT alone instead of silently evaluating someone
+ * else's predicate against an INCLUDE column.
*/
- private void extractFieldNames(ILogicalExpression expr, ARecordType recordType, Set<String> fieldNames) {
+ private void extractFieldPaths(ILogicalExpression expr, VectorSearchInfo searchInfo, Set<List<String>> fieldPaths)
+ throws AlgebricksException {
if (expr.getExpressionTag() != LogicalExpressionTag.FUNCTION_CALL) {
return;
}
AbstractFunctionCallExpression funcExpr = (AbstractFunctionCallExpression) expr;
+ List<String> path = resolveFieldPath(funcExpr, searchInfo);
+ if (path != null) {
+ fieldPaths.add(path);
+ // Do not recurse: the nested accesses under this one are the prefix of the path just added.
+ return;
+ }
+
+ // Recurse into arguments
+ for (Mutable<ILogicalExpression> arg : funcExpr.getArguments()) {
+ extractFieldPaths(arg.getValue(), searchInfo, fieldPaths);
+ }
+ }
+
+ /**
+ * Resolve a (possibly nested) field-access expression to its full path relative to one of this vector
+ * search's record variables, or {@code null} if it is not such an access.
+ * <p>
+ * {@code FIELD_ACCESS_BY_INDEX}'s integer is an index into the field names of the type of ITS OWN base
+ * expression, so for a nested access the base's type has to be resolved first (descending the record
+ * type as the recursion descends) — resolving every level against the top-level record type produces a
+ * bogus name, which in the worst case collides with a real INCLUDE path. Note that
+ * {@code ByNameToByIndexFieldAccessRule} has already converted accesses on a declared type to
+ * {@code FIELD_ACCESS_BY_INDEX} by the time this rule runs, so the nested case is the common one.
+ */
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.GENERATED)
+ private List<String> resolveFieldPath(AbstractFunctionCallExpression funcExpr, VectorSearchInfo searchInfo)
+ throws AlgebricksException {
FunctionIdentifier fid = funcExpr.getFunctionIdentifier();
+ boolean byName = fid.equals(BuiltinFunctions.FIELD_ACCESS_BY_NAME);
+ boolean byIndex = fid.equals(BuiltinFunctions.FIELD_ACCESS_BY_INDEX);
+ if ((!byName && !byIndex) || funcExpr.getArguments().size() < 2) {
+ return null;
+ }
- if (fid.equals(BuiltinFunctions.FIELD_ACCESS_BY_NAME)) {
- if (funcExpr.getArguments().size() >= 2) {
- String fieldName = AccessMethodUtils.getStringConstant(funcExpr.getArguments().get(1));
- if (fieldName != null) {
- fieldNames.add(fieldName);
- }
+ // Resolve the base: either this search's record variable (empty prefix) or an enclosing field access.
+ ILogicalExpression base = funcExpr.getArguments().get(0).getValue();
+ List<String> prefix;
+ if (base.getExpressionTag() == LogicalExpressionTag.VARIABLE) {
+ LogicalVariable baseVar = ((VariableReferenceExpression) base).getVariableReference();
+ if (!searchInfo.recordVars().contains(baseVar)) {
+ return null;
}
- } else if (fid.equals(BuiltinFunctions.FIELD_ACCESS_BY_INDEX)) {
- if (funcExpr.getArguments().size() >= 2 && recordType != null) {
- Integer idx = AccessMethodUtils.getInt32Constant(funcExpr.getArguments().get(1));
- if (idx != null) {
- String[] names = recordType.getFieldNames();
- if (idx >= 0 && idx < names.length) {
- fieldNames.add(names[idx]);
- }
- }
+ prefix = List.of();
+ } else if (base.getExpressionTag() == LogicalExpressionTag.FUNCTION_CALL) {
+ prefix = resolveFieldPath((AbstractFunctionCallExpression) base, searchInfo);
+ if (prefix == null) {
+ return null;
}
+ } else {
+ return null;
}
- // Recurse into arguments
- for (Mutable<ILogicalExpression> arg : funcExpr.getArguments()) {
- extractFieldNames(arg.getValue(), recordType, fieldNames);
+ String fieldName;
+ if (byName) {
+ fieldName = AccessMethodUtils.getStringConstant(funcExpr.getArguments().get(1));
+ } else {
+ Integer idx = AccessMethodUtils.getInt32Constant(funcExpr.getArguments().get(1));
+ ARecordType baseType = resolveRecordTypeAt(searchInfo.recordType(), prefix);
+ if (idx == null || baseType == null) {
+ return null;
+ }
+ String[] names = baseType.getFieldNames();
+ fieldName = idx >= 0 && idx < names.length ? names[idx] : null;
+ }
+ if (fieldName == null) {
+ return null;
}
+
+ List<String> path = new ArrayList<>(prefix);
+ path.add(fieldName);
+ return path;
+ }
+
+ /**
+ * The record type reached by following {@code path} from {@code recordType}, or {@code null} if the path
+ * does not resolve to a record type (open field, non-record type, unknown path).
+ */
+ private ARecordType resolveRecordTypeAt(ARecordType recordType, List<String> path) throws AlgebricksException {
+ if (recordType == null) {
+ return null;
+ }
+ if (path.isEmpty()) {
+ return recordType;
+ }
+ IAType type = recordType.getSubFieldType(path);
+ if (type != null && type.getTypeTag() == ATypeTag.OBJECT) {
+ return (ARecordType) type;
+ }
+ return null;
}
/**
* Rewrites field-access expressions to use new INCLUDE field variables.
* Example: gt($row.getField(2), 2000) -> gt($year, 2000)
*/
- private ILogicalExpression rewriteFieldAccess(ILogicalExpression expr, Map<String, LogicalVariable> fieldToVar,
- ARecordType recordType) {
+ private ILogicalExpression rewriteFieldAccess(ILogicalExpression expr,
+ Map<List<String>, LogicalVariable> fieldToVar, VectorSearchInfo searchInfo) throws AlgebricksException {
if (expr.getExpressionTag() != LogicalExpressionTag.FUNCTION_CALL) {
return expr;
}
AbstractFunctionCallExpression funcExpr = (AbstractFunctionCallExpression) expr;
- FunctionIdentifier fid = funcExpr.getFunctionIdentifier();
-
- // Check if this is a field access to replace
- String fieldName = null;
-
- if (fid.equals(BuiltinFunctions.FIELD_ACCESS_BY_NAME)) {
- if (funcExpr.getArguments().size() >= 2) {
- fieldName = AccessMethodUtils.getStringConstant(funcExpr.getArguments().get(1));
- }
- } else if (fid.equals(BuiltinFunctions.FIELD_ACCESS_BY_INDEX)) {
- if (funcExpr.getArguments().size() >= 2 && recordType != null) {
- Integer idx = AccessMethodUtils.getInt32Constant(funcExpr.getArguments().get(1));
- if (idx != null) {
- String[] names = recordType.getFieldNames();
- if (idx >= 0 && idx < names.length) {
- fieldName = names[idx];
- }
- }
- }
- }
- // Replace field access with variable reference
- if (fieldName != null && fieldToVar.containsKey(fieldName)) {
- LogicalVariable newVar = fieldToVar.get(fieldName);
+ // Replace a field access with a variable reference only when its FULL path — resolved against this
+ // search's record variable — is one of the INCLUDE fields we created a variable for.
+ List<String> path = resolveFieldPath(funcExpr, searchInfo);
+ if (path != null && fieldToVar.containsKey(path)) {
+ LogicalVariable newVar = fieldToVar.get(path);
VariableReferenceExpression varRef = new VariableReferenceExpression(newVar);
varRef.setSourceLocation(funcExpr.getSourceLocation());
return varRef;
@@ -455,7 +518,7 @@ public class PushFilterIntoVectorSearchRule implements IAlgebraicRewriteRule {
boolean changed = false;
for (Mutable<ILogicalExpression> argRef : funcExpr.getArguments()) {
- ILogicalExpression newArg = rewriteFieldAccess(argRef.getValue(), fieldToVar, recordType);
+ ILogicalExpression newArg = rewriteFieldAccess(argRef.getValue(), fieldToVar, searchInfo);
newArgs.add(new MutableObject<>(newArg));
if (newArg != argRef.getValue()) {
changed = true;
diff --git a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IntroduceTopKAccessMethodRule.java b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IntroduceTopKAccessMethodRule.java
index 56736fb194..e455772f70 100644
--- a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IntroduceTopKAccessMethodRule.java
+++ b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IntroduceTopKAccessMethodRule.java
@@ -712,9 +712,14 @@ public class IntroduceTopKAccessMethodRule extends AbstractIntroduceAccessMethod
fieldMatch = new Pair<>(VectorIndexAccessMethod.INSTANCE, index);
}
} else {
- // No query metric available - use first field match (backward compatibility).
- result.add(new Pair<>(VectorIndexAccessMethod.INSTANCE, index));
- break;
+ // No query metric available. Do NOT pick an arbitrary index here: without a metric to
+ // compare against, the first VTREE index found could easily answer a cosine query from a
+ // euclidean index — silently wrong results. This should be unreachable (isAnnDistance()
+ // requires the ANN hint and the rewrite visitor always fills in a validated metric), so
+ // bail out to the exact plan rather than guess.
+ LOGGER.warn("No query distance metric available for ANN search; skipping vector index "
+ + "selection and falling back to full scan (KNN).");
+ return;
}
}
}
diff --git a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/VectorIndexAccessMethod.java b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/VectorIndexAccessMethod.java
index 4d27d74063..503c74c334 100644
--- a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/VectorIndexAccessMethod.java
+++ b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/VectorIndexAccessMethod.java
@@ -297,7 +297,18 @@ public class VectorIndexAccessMethod implements IAccessMethod {
// Index-only opportunity detected upstream in IntroduceTopKAccessMethodRule (PK-only
// projection above LIMIT). When set, the secondary UnnestMap emits an extra $$dist field, the
// ORDER BY rebinds to it, and the primary BTree lookup + rerank are skipped (see branch below).
+ //
+ // Every precondition must be settled HERE, before the params are handed to
+ // AccessMethodUtils.createSecondaryIndexUnnestMap(): that call serializes jobGenParams into the
+ // index-search function arguments, so clearing indexOnly afterwards does not change the emitted plan
+ // — the runtime would still emit [pk..., dist] while the plan declares only the PK variables, an
+ // output arity mismatch. The ORDER shape the branch below relies on is checkable now.
boolean isIndexOnlyPlan = indexOnly;
+ if (isIndexOnlyPlan) {
+ ILogicalOperator orderCandidate = orderRef.getValue();
+ isIndexOnlyPlan = orderCandidate.getOperatorTag() == LogicalOperatorTag.ORDER
+ && ((OrderOperator) orderCandidate).getOrderExpressions().size() == 1;
+ }
jobGenParams.setIndexOnly(isIndexOnlyPlan);
// Create UNNEST-MAP operator for vector index search
@@ -329,120 +340,113 @@ public class VectorIndexAccessMethod implements IAccessMethod {
// 5. Skip createRestOfIndexSearchPlan — no primary BTree lookup, no record assembly. The
// runtime reads D(q,x) from the cursor and appends it.
if (!(secondaryIndexUnnestOp instanceof UnnestMapOperator)) {
- // Unexpected shape — fall through to the legacy lookup-and-rerank plan below.
- isIndexOnlyPlan = false;
- jobGenParams.setIndexOnly(false);
+ // Not recoverable: indexOnly=true is already baked into the index-search arguments above, so
+ // there is no falling back to lookup-and-rerank from here. retainInput/retainNull are both
+ // false, which is exactly the case where createSecondaryIndexUnnestMap returns an
+ // UnnestMapOperator, so this is a broken invariant rather than an unsupported plan shape.
+ throw new AlgebricksException("Vector index-only plan: expected an UnnestMapOperator from the "
+ + "secondary index search but got " + secondaryIndexUnnestOp.getOperatorTag());
} else {
UnnestMapOperator unnestMap = (UnnestMapOperator) secondaryIndexUnnestOp;
LogicalVariable distVar = context.newVar();
unnestMap.getVariables().add(distVar);
unnestMap.getVariableTypes().add(BuiltinType.ADOUBLE);
- // Rewrite ORDER BY: replace the ann_distance call with VarRef($$dist). The earlier
- // pattern-match guarantees orderOp has exactly one expression (the ann_distance call);
- // keep the same direction.
+ // Rewrite ORDER BY: replace the ann_distance call with VarRef($$dist). isIndexOnlyPlan was
+ // only left true if orderOp has exactly one expression (the ann_distance call); keep the same
+ // direction.
OrderOperator orderOp = (OrderOperator) orderRef.getValue();
- if (orderOp.getOrderExpressions().size() != 1) {
- // Unexpected ORDER shape — roll back the distVar addition before falling through.
- unnestMap.getVariables().remove(unnestMap.getVariables().size() - 1);
- unnestMap.getVariableTypes().remove(unnestMap.getVariableTypes().size() - 1);
- isIndexOnlyPlan = false;
- jobGenParams.setIndexOnly(false);
- } else {
- VariableReferenceExpression distVarRef = new VariableReferenceExpression(distVar);
- distVarRef.setSourceLocation(orderOp.getSourceLocation());
- orderOp.getOrderExpressions().get(0).second.setValue(distVarRef);
+ VariableReferenceExpression distVarRef = new VariableReferenceExpression(distVar);
+ distVarRef.setSourceLocation(orderOp.getSourceLocation());
+ orderOp.getOrderExpressions().get(0).second.setValue(distVarRef);
+
+ // Build the variable substitution map: old PK vars → new PK vars. The secondary
+ // UnnestMap allocates fresh context.newVar() PK variables, so anything that
+ // referenced the old PK variable from the DataSourceScan needs to be redirected.
+ List<LogicalVariable> oldVars = dataSourceOp.getVariables();
+ int numPK = dataset.getPrimaryKeys().size();
+ List<LogicalVariable> newPkVars = unnestMap.getVariables().subList(0, numPK);
+ Map<LogicalVariable, LogicalVariable> pkSubstitution = new java.util.HashMap<>();
+ for (int i = 0; i < numPK; i++) {
+ pkSubstitution.put(oldVars.get(i), newPkVars.get(i));
}
-
- if (isIndexOnlyPlan) {
- // Build the variable substitution map: old PK vars → new PK vars. The secondary
- // UnnestMap allocates fresh context.newVar() PK variables, so anything that
- // referenced the old PK variable from the DataSourceScan needs to be redirected.
- List<LogicalVariable> oldVars = dataSourceOp.getVariables();
- int numPK = dataset.getPrimaryKeys().size();
- List<LogicalVariable> newPkVars = unnestMap.getVariables().subList(0, numPK);
- Map<LogicalVariable, LogicalVariable> pkSubstitution = new java.util.HashMap<>();
- for (int i = 0; i < numPK; i++) {
- pkSubstitution.put(oldVars.get(i), newPkVars.get(i));
- }
- // The record variable (and meta record var, if present) — we rewrite
- // field-access($$rec, "<pk_field>") to VarRef($$pk_new) below.
- LogicalVariable oldRecVar = numPK < oldVars.size() ? oldVars.get(numPK) : null;
-
- // Map PK field name -> new PK variable.
- Map<String, LogicalVariable> pkFieldToNewVar = new java.util.HashMap<>();
- List<List<String>> pkPaths = dataset.getPrimaryKeys();
- for (int i = 0; i < numPK; i++) {
- List<String> p = pkPaths.get(i);
- if (p != null && p.size() == 1) {
- pkFieldToNewVar.put(p.get(0), newPkVars.get(i));
- }
+ // The record variable (and meta record var, if present) — we rewrite
+ // field-access($$rec, "<pk_field>") to VarRef($$pk_new) below.
+ LogicalVariable oldRecVar = numPK < oldVars.size() ? oldVars.get(numPK) : null;
+
+ // Map PK field name -> new PK variable.
+ Map<String, LogicalVariable> pkFieldToNewVar = new java.util.HashMap<>();
+ List<List<String>> pkPaths = dataset.getPrimaryKeys();
+ for (int i = 0; i < numPK; i++) {
+ List<String> p = pkPaths.get(i);
+ if (p != null && p.size() == 1) {
+ pkFieldToNewVar.put(p.get(0), newPkVars.get(i));
}
+ }
- // The rewrites below must cover the operators ABOVE the LIMIT too — the result
- // projection (e.g. SELECT VALUE m.idx) lives above LIMIT and references the old
- // record var via field-access($$rec, "idx"). Walking only from limitRef would leave
- // it dangling ("Could not infer type"). aboveLimitOps[0] is the outermost ancestor
- // (e.g. DistributeResult); walking from it reaches the projection, the LIMIT, the
- // ORDER, and the ASSIGNs below in a single descent.
- ILogicalOperator rewriteRoot = (aboveLimitOps != null && !aboveLimitOps.isEmpty())
- ? aboveLimitOps.get(0) : (ILogicalOperator) limitRef.getValue();
-
- // 1. Substitute old PK var refs throughout the plan (above and below LIMIT).
- org.apache.hyracks.algebricks.core.algebra.operators.logical.visitors.VariableUtilities
- .substituteVariablesInDescendantsAndSelf(rewriteRoot, pkSubstitution, context);
-
- // 2. Rewrite field-access on the old record var to direct PK VarRefs (e.g. the
- // SELECT VALUE m.idx projection above LIMIT).
- if (oldRecVar != null) {
- rewriteRecordFieldAccessToPk(rewriteRoot, oldRecVar, pkFieldToNewVar);
- }
+ // The rewrites below must cover the operators ABOVE the LIMIT too — the result
+ // projection (e.g. SELECT VALUE m.idx) lives above LIMIT and references the old
+ // record var via field-access($$rec, "idx"). Walking only from limitRef would leave
+ // it dangling ("Could not infer type"). aboveLimitOps[0] is the outermost ancestor
+ // (e.g. DistributeResult); walking from it reaches the projection, the LIMIT, the
+ // ORDER, and the ASSIGNs below in a single descent.
+ ILogicalOperator rewriteRoot = (aboveLimitOps != null && !aboveLimitOps.isEmpty())
+ ? aboveLimitOps.get(0) : (ILogicalOperator) limitRef.getValue();
+
+ // 1. Substitute old PK var refs throughout the plan (above and below LIMIT).
+ org.apache.hyracks.algebricks.core.algebra.operators.logical.visitors.VariableUtilities
+ .substituteVariablesInDescendantsAndSelf(rewriteRoot, pkSubstitution, context);
+
+ // 2. Rewrite field-access on the old record var to direct PK VarRefs (e.g. the
+ // SELECT VALUE m.idx projection above LIMIT).
+ if (oldRecVar != null) {
+ rewriteRecordFieldAccessToPk(rewriteRoot, oldRecVar, pkFieldToNewVar);
+ }
- // 3. Any remaining ASSIGN expression that still references oldRecVar (transitively) —
- // for example $$x := $$m.getField("embedding") hoisted by the SQL++ compiler out of
- // ann_distance(m.embedding, ...) — is now structurally orphaned: no operator above
- // LIMIT consumes it (we wouldn't have entered the index-only branch otherwise) but the
- // type system still walks it. Replace each such expression with MISSING so it types
- // cleanly. None of these values is ever read at runtime.
- if (oldRecVar != null) {
- neutralizeDanglingExpressions(rewriteRoot, oldRecVar);
- }
+ // 3. Any remaining ASSIGN expression that still references oldRecVar (transitively) —
+ // for example $$x := $$m.getField("embedding") hoisted by the SQL++ compiler out of
+ // ann_distance(m.embedding, ...) — is now structurally orphaned: no operator above
+ // LIMIT consumes it (we wouldn't have entered the index-only branch otherwise) but the
+ // type system still walks it. Replace each such expression with MISSING so it types
+ // cleanly. None of these values is ever read at runtime.
+ if (oldRecVar != null) {
+ neutralizeDanglingExpressions(rewriteRoot, oldRecVar);
+ }
- // Recompute type env bottom-up over every operator whose schema/expressions changed:
- // the new secondary UnnestMap, the LIMIT chain, then each above-LIMIT ancestor from the
- // one closest to LIMIT up to the root (aboveLimitOps is ordered root-first).
- context.computeAndSetTypeEnvironmentForOperator(secondaryIndexUnnestOp);
- org.apache.hyracks.algebricks.core.algebra.util.OperatorPropertiesUtil.typeOpRec(limitRef, context);
- if (aboveLimitOps != null) {
- for (int ai = aboveLimitOps.size() - 1; ai >= 0; ai--) {
- context.computeAndSetTypeEnvironmentForOperator(aboveLimitOps.get(ai));
- }
+ // Recompute type env bottom-up over every operator whose schema/expressions changed:
+ // the new secondary UnnestMap, the LIMIT chain, then each above-LIMIT ancestor from the
+ // one closest to LIMIT up to the root (aboveLimitOps is ordered root-first).
+ context.computeAndSetTypeEnvironmentForOperator(secondaryIndexUnnestOp);
+ org.apache.hyracks.algebricks.core.algebra.util.OperatorPropertiesUtil.typeOpRec(limitRef, context);
+ if (aboveLimitOps != null) {
+ for (int ai = aboveLimitOps.size() - 1; ai >= 0; ai--) {
+ context.computeAndSetTypeEnvironmentForOperator(aboveLimitOps.get(ai));
}
+ }
- // Cross-pollination dedup (index-only branch): when cross_pollination_m > 1 the secondary
- // cursor emits up to M (pk..., dist) copies per record. The primary-lookup path dedups via a
- // DistinctOperator above its primary UNNEST-MAP (see createRestOfIndexSearchPlan below), but the
- // index-only branch skips that path, so it must splice its OWN Distinct — keyed on the new
- // secondary-UnnestMap PK vars — directly above the secondary UNNEST-MAP and below ORDER BY/LIMIT,
- // so LIMIT applies to DISTINCT PKs. DistinctOperator propagates all input vars, so $$dist
- // survives for the ORDER BY $$dist above it. For M == 1 we return the bare UNNEST-MAP so the
- // plan stays byte-identical to the legacy path.
- if (extractCrossPollinationM(chosenIndex) > 1 && numPK > 0) {
- List<Mutable<ILogicalExpression>> dedupExprs = new ArrayList<>(numPK);
- for (LogicalVariable pkVar : newPkVars) {
- VariableReferenceExpression pkRef = new VariableReferenceExpression(pkVar);
- pkRef.setSourceLocation(secondaryIndexUnnestOp.getSourceLocation());
- dedupExprs.add(new MutableObject<>(pkRef));
- }
- DistinctOperator dedupOp = new DistinctOperator(dedupExprs);
- dedupOp.setSourceLocation(secondaryIndexUnnestOp.getSourceLocation());
- dedupOp.getInputs().add(new MutableObject<>(secondaryIndexUnnestOp));
- dedupOp.setExecutionMode(secondaryIndexUnnestOp.getExecutionMode());
- context.computeAndSetTypeEnvironmentForOperator(dedupOp);
- return dedupOp;
+ // Cross-pollination dedup (index-only branch): when cross_pollination_m > 1 the secondary
+ // cursor emits up to M (pk..., dist) copies per record. The primary-lookup path dedups via a
+ // DistinctOperator above its primary UNNEST-MAP (see createRestOfIndexSearchPlan below), but the
+ // index-only branch skips that path, so it must splice its OWN Distinct — keyed on the new
+ // secondary-UnnestMap PK vars — directly above the secondary UNNEST-MAP and below ORDER BY/LIMIT,
+ // so LIMIT applies to DISTINCT PKs. DistinctOperator propagates all input vars, so $$dist
+ // survives for the ORDER BY $$dist above it. For M == 1 we return the bare UNNEST-MAP so the
+ // plan stays byte-identical to the legacy path.
+ if (extractCrossPollinationM(chosenIndex) > 1 && numPK > 0) {
+ List<Mutable<ILogicalExpression>> dedupExprs = new ArrayList<>(numPK);
+ for (LogicalVariable pkVar : newPkVars) {
+ VariableReferenceExpression pkRef = new VariableReferenceExpression(pkVar);
+ pkRef.setSourceLocation(secondaryIndexUnnestOp.getSourceLocation());
+ dedupExprs.add(new MutableObject<>(pkRef));
}
- return secondaryIndexUnnestOp;
+ DistinctOperator dedupOp = new DistinctOperator(dedupExprs);
+ dedupOp.setSourceLocation(secondaryIndexUnnestOp.getSourceLocation());
+ dedupOp.getInputs().add(new MutableObject<>(secondaryIndexUnnestOp));
+ dedupOp.setExecutionMode(secondaryIndexUnnestOp.getExecutionMode());
+ context.computeAndSetTypeEnvironmentForOperator(dedupOp);
+ return dedupOp;
}
+ return secondaryIndexUnnestOp;
}
}
diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
index 3a004a6ca9..c9ddc29790 100644
--- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
+++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
@@ -1760,6 +1760,25 @@ public class QueryTranslator extends AbstractLangTranslator implements IStatemen
CreateIndexStatement.IndexedElement indexedElement = stmtCreateIndex.getIndexedElements().get(0);
List<String> keyFieldNames = indexedElement.getProjectList().get(0).first;
+ // The vector field is not part of `indexedElements` for a VTREE (that local was swapped to the
+ // INCLUDE list above), so the shared "cannot create index on meta fields" check did not see
+ // it. Reject a meta-sourced vector field here: IndexTupleTranslator does not persist source
+ // indicators for a vector index — it hardcodes RECORD_INDICATOR on read — so a meta-sourced
+ // field would work in the session that created it and silently switch to record-sourced once
+ // the metadata is re-read.
+ if (indexedElement.getSourceIndicator() != Index.RECORD_INDICATOR) {
+ throw new AsterixException(ErrorCode.COMPILATION_ERROR, indexedElement.getSourceLocation(),
+ "Cannot create index on meta fields");
+ }
+
+ // LSM component filters are not applied by the vector index (MetadataProvider drops the filter
+ // factories and VTree.getNumOfFilterFields() is 0). Creating one on a filtered dataset would
+ // silently make the user's filter a no-op for this index, so reject the combination instead.
+ if (DatasetUtil.getFilterField(ds) != null) {
+ throw new CompilationException(ErrorCode.COMPILATION_ERROR, sourceLoc,
+ "Cannot create a vector index on a dataset with a filter field");
+ }
+
// Validate that the vector field exists in the dataset schema
Triple<IAType, Boolean, Boolean> vectorFieldType =
KeyFieldTypeUtil.getKeyProjectType(aRecordType, keyFieldNames, sourceLoc);
diff --git a/asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj b/asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj
index 6ad1db3e56..dc1621dd12 100644
--- a/asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj
+++ b/asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj
@@ -1578,7 +1578,7 @@ CreateIndexStatement IndexSpecification(Token startStmtToken) throws ParseExcept
String fullTextConfigName = null;
Token startElementToken = null;
Boolean excludeUnknown = null;
- Pair<Map<String, String>, Boolean> castConfigDefaultNull;
+ Pair<Map<String, String>, Boolean> castConfigDefaultNull = null;
Boolean castDefaultNull = null;
Map<String, String> castConfig = null;
RecordConstructor withRecord = null;
@@ -1702,6 +1702,17 @@ CreateIndexStatement IndexSpecification(Token startStmtToken) throws ParseExcept
"Vector index requires WITH clause specifying index configuration");
}
+ // ENFORCED and CAST(...) do not apply to a vector index: the statement below hardcodes them to
+ // false/null, so accepting them would silently discard what the user typed.
+ if (enforced) {
+ throw new SqlppParseException(getSourceLocation(startStmtToken),
+ "ENFORCED is not supported for a vector index (TYPE VTREE)");
+ }
+ if (castConfigDefaultNull != null) {
+ throw new SqlppParseException(getSourceLocation(startStmtToken),
+ "CAST (...) is not supported for a vector index (TYPE VTREE)");
+ }
+
// Validate that EXCLUDE UNKNOWN KEY is specified
if (excludeUnknown == null || !excludeUnknown) {
throw new SqlppParseException(getSourceLocation(startStmtToken),
@@ -1729,6 +1740,13 @@ CreateIndexStatement IndexSpecification(Token startStmtToken) throws ParseExcept
"INCLUDE clause is only supported for vector indexes (TYPE VTREE)");
}
+ // Validate: the WITH clause is only for vector indexes. The non-vector statement built below does not
+ // read withRecord, so accepting it here would silently discard it (e.g. a typo'd option on a BTREE).
+ if (withRecord != null) {
+ throw new SqlppParseException(getSourceLocation(startStmtToken),
+ "WITH clause is only supported for vector indexes (TYPE VTREE)");
+ }
+
// Handle regular (non-vector) index creation
IndexType indexType;
int gramLength;
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java
index 3e58295934..a17bc78cbf 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java
@@ -1682,14 +1682,17 @@ public class MetadataProvider implements IMetadataProvider<DataSourceId, String>
Dataset dataset = MetadataManagerUtil.findExistingDataset(mdTxnCtx, database, dataverseName, datasetName);
int numKeys = primaryKeys.size() + secondaryKeys.size();
- int numFilterFields = DatasetUtil.getFilterField(dataset) == null ? 0 : 1;
- // Vector indexes do not support tuple filters; drop any factories supplied by the caller.
+ // Vector indexes do not support tuple filters; drop any factories supplied by the caller, and do NOT
+ // append a filter column to the permutation either — a filter field the index never reads would still
+ // cost bytes in every data tuple (VTree.getNumOfFilterFields() is 0 on the storage side). CREATE INDEX
+ // rejects a vector index on a filtered dataset (see QueryTranslator), so this is belt-and-braces for
+ // an index created before that check existed.
filterFactory = null;
prevFilterFactory = null;
- // Field permutation layout: <secondary keys (vector + include fields), primary keys, filter field?>
- int[] fieldPermutation = new int[numKeys + numFilterFields];
+ // Field permutation layout: <secondary keys (vector + include fields), primary keys>
+ int[] fieldPermutation = new int[numKeys];
int[] modificationCallbackPrimaryKeyFields = new int[primaryKeys.size()];
int[] pkFields = new int[primaryKeys.size()];
int i = 0;
@@ -1710,15 +1713,10 @@ public class MetadataProvider implements IMetadataProvider<DataSourceId, String>
j++;
}
- if (numFilterFields > 0) {
- int idx = propagatedSchema.findVariable(additionalNonKeyFields.get(0));
- fieldPermutation[numKeys] = idx;
- }
-
// For UPSERT, also build the permutation for the previous tuple.
int[] prevFieldPermutation = null;
if (indexOp == IndexOperation.UPSERT) {
- prevFieldPermutation = new int[numKeys + numFilterFields];
+ prevFieldPermutation = new int[numKeys];
int k = 0;
for (LogicalVariable varKey : prevSecondaryKeys) {
@@ -1732,11 +1730,6 @@ public class MetadataProvider implements IMetadataProvider<DataSourceId, String>
prevFieldPermutation[k] = idx;
k++;
}
-
- if (numFilterFields > 0) {
- int idx = propagatedSchema.findVariable(prevAdditionalFilteringKeys.get(0));
- prevFieldPermutation[numKeys] = idx;
- }
}
try {
diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryVectorOperationsHelper.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryVectorOperationsHelper.java
index 55737dcc4a..d3b99a5675 100644
--- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryVectorOperationsHelper.java
+++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryVectorOperationsHelper.java
@@ -109,6 +109,7 @@ import org.apache.hyracks.storage.am.vector.api.IVTreeBinaryAccessorFactory;
import org.apache.hyracks.storage.common.IResourceFactory;
import org.apache.hyracks.storage.common.IStorageManager;
import org.apache.hyracks.storage.common.projection.ITupleProjectorFactory;
+import org.apache.hyracks.util.annotations.AiProvenance;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -207,6 +208,25 @@ public class SecondaryVectorOperationsHelper extends SecondaryTreeIndexOperation
return sampleSize < TRAIN_LIST_MIN_SAMPLE_SIZE;
}
+ /**
+ * The seed shared by the train-list sample and the k-means RNG: the request-level
+ * {@code compiler.vector.trainseed} when set (so an index build is reproducible), otherwise a fresh
+ * {@code nanoTime()} seed.
+ */
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.GENERATED)
+ private long resolveTrainSeed() {
+ Object trainSeedCfg = metadataProvider.getConfig().get(CompilerProperties.COMPILER_VECTOR_TRAINSEED_KEY);
+ if (trainSeedCfg != null) {
+ try {
+ return Long.parseLong(String.valueOf(trainSeedCfg).trim());
+ } catch (NumberFormatException e) {
+ LOGGER.warn("Invalid {} '{}', using a random seed", CompilerProperties.COMPILER_VECTOR_TRAINSEED_KEY,
+ trainSeedCfg);
+ }
+ }
+ return System.nanoTime();
+ }
+
@Override
public JobSpecification buildStaticStructureJobSpec() throws AlgebricksException {
IDataFormat format = metadataProvider.getDataFormat();
@@ -236,9 +256,14 @@ public class SecondaryVectorOperationsHelper extends SecondaryTreeIndexOperation
AdmObjectNode withObjectNodeForSampling = indexDetails.getWithObjectNode();
double trainListFraction = (withObjectNodeForSampling != null)
? withObjectNodeForSampling.getOptionalDouble("train_list_fraction", 0.1) : 0.1;
- // sample_seed is not a supported WITH field (not in VectorIndexDeclUtil.ALLOWED_VECTOR_INDEX_WITH_FIELDS),
- // so it was never actually settable; deterministic training is driven by compiler.vector.trainseed instead.
- long sampleSeed = System.currentTimeMillis();
+ // Seed for BOTH the train-list sample and the k-means RNG: overridable per request
+ // (SET `compiler.vector.trainseed` "42") so CI / regression tests get reproducible centroids, and a
+ // fresh random seed otherwise. The sample seed must honour the same override, otherwise a sampled
+ // train list (train_list_fraction < 1.0) makes the whole index build irreproducible no matter what
+ // the k-means seed is. sample_seed is not a supported WITH field
+ // (see VectorIndexDeclUtil.ALLOWED_VECTOR_INDEX_WITH_FIELDS), so this is the only channel.
+ long trainSeed = resolveTrainSeed();
+ long sampleSeed = trainSeed;
// Retrieve cardinality from sample index metadata (needed for fraction-based sample size)
Index sampleIndex = metadataProvider.findSampleIndex(dataset.getDatabaseName(), dataset.getDataverseName(),
@@ -288,13 +313,15 @@ public class SecondaryVectorOperationsHelper extends SecondaryTreeIndexOperation
UUID sampleUUID = UUID.randomUUID();
UUID tupleCountUUID = UUID.randomUUID();
- // Extract K value from WITH clause
+ // Extract K value from WITH clause.
+ // The default is sqrt(cardinality / partitions), clamped to at least 1: for a small dataset on a
+ // multi-partition cluster (datasetCardinality < numPartitions) the expression is 0, and hierarchical
+ // k-means cannot be run with K = 0. An explicit num_clusters is already validated as >= 1 by
+ // VectorIndexDeclUtil.validateNumClusters, so the clamp only affects the computed default.
AdmObjectNode withObjectNode = indexDetails.getWithObjectNode();
- int K = (int) Math.sqrt((double) datasetCardinality / numPartitions); // default value
- if (withObjectNode != null) {
- K = withObjectNode.getOptionalInt("num_clusters",
- (int) Math.sqrt((double) datasetCardinality / numPartitions));
- }
+ int defaultNumClusters = Math.max(1, (int) Math.sqrt((double) datasetCardinality / numPartitions));
+ int K = withObjectNode != null ? withObjectNode.getOptionalInt("num_clusters", defaultNumClusters)
+ : defaultNumClusters;
// Distance metric from index DDL (WITH similarity "euclidean"|"cosine"|"cosine similarity"|etc.).
// For cosine, embeddings must be L2-normalized to unit length before insert; the engine does not normalize.
@@ -328,19 +355,6 @@ public class SecondaryVectorOperationsHelper extends SecondaryTreeIndexOperation
// ====== STATIC STRUCTURE JOB: K-MEANS → STATIC STRUCTURE CREATION ======
- // Training RNG seed: overridable per request (SET `compiler.vector.trainseed` "42") so CI /
- // regression tests get reproducible centroids; defaults to a fresh nanoTime seed otherwise.
- long trainSeed = System.nanoTime();
- Object trainSeedCfg = metadataProvider.getConfig().get(CompilerProperties.COMPILER_VECTOR_TRAINSEED_KEY);
- if (trainSeedCfg != null) {
- try {
- trainSeed = Long.parseLong(String.valueOf(trainSeedCfg).trim());
- } catch (NumberFormatException e) {
- LOGGER.warn("Invalid {} '{}', using a random seed", CompilerProperties.COMPILER_VECTOR_TRAINSEED_KEY,
- trainSeedCfg);
- }
- }
-
HierarchicalKMeansPlusPlusCentroidsOperatorDescriptor candidates =
new HierarchicalKMeansPlusPlusCentroidsOperatorDescriptor(spec, hierarchicalRecDesc, secondaryRecDesc,
sampleUUID, tupleCountUUID, new ColumnAccessEvalFactory(0), K, maxScalableKmeansIter,
diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractLSMIndex.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractLSMIndex.java
index 0609ef2382..c84dda638e 100644
--- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractLSMIndex.java
+++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractLSMIndex.java
@@ -236,7 +236,12 @@ public abstract class AbstractLSMIndex implements ILSMIndex {
protected void completeActivation() throws HyracksDataException {
}
- private void loadDiskComponents() throws HyracksDataException {
+ /**
+ * Load this index's disk components from the valid on-disk files. Called from {@link #activate()}.
+ * Subclasses whose components need extra loading steps (e.g. a shared sidecar component) override this
+ * and delegate to {@code super.loadDiskComponents()}.
+ */
+ protected void loadDiskComponents() throws HyracksDataException {
diskComponents.clear();
List<LSMComponentFileReferences> validFileReferences = fileManager.cleanupAndGetValidFiles();
for (LSMComponentFileReferences lsmComponentFileReferences : validFileReferences) {
diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/VTreeSearchOperatorNodePushable.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/VTreeSearchOperatorNodePushable.java
index 6e0d5f7091..849a869a16 100644
--- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/VTreeSearchOperatorNodePushable.java
+++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/VTreeSearchOperatorNodePushable.java
@@ -26,6 +26,8 @@ import org.apache.hyracks.api.exceptions.HyracksDataException;
import org.apache.hyracks.api.util.HyracksConstants;
import org.apache.hyracks.data.std.primitive.DoublePointable;
import org.apache.hyracks.data.std.primitive.IntegerPointable;
+import org.apache.hyracks.data.std.primitive.LongPointable;
+import org.apache.hyracks.data.std.primitive.ShortPointable;
import org.apache.hyracks.dataflow.common.data.accessors.ITupleReference;
import org.apache.hyracks.dataflow.common.data.accessors.PermutingFrameTupleReference;
import org.apache.hyracks.storage.am.common.api.ISearchOperationCallbackFactory;
@@ -44,6 +46,7 @@ import org.apache.hyracks.storage.common.IIndexAccessParameters;
import org.apache.hyracks.storage.common.IIndexCursor;
import org.apache.hyracks.storage.common.ISearchPredicate;
import org.apache.hyracks.storage.common.projection.ITupleProjectorFactory;
+import org.apache.hyracks.util.annotations.AiProvenance;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -184,6 +187,14 @@ public class VTreeSearchOperatorNodePushable extends IndexSearchOperatorNodePush
private static final int QP_FIELD_MIN_PROBE_FRACTION = 3;
private static final int QP_FIELD_K_MULTIPLIER = 4;
+ // Serialized ADM integer type tags, matching ATypeTag.{TINYINT,SMALLINT,INTEGER,BIGINT}.serialize() in
+ // asterix-om. Hardcoded for the same reason as ADOUBLE_TYPE_TAG below: Hyracks cannot depend on
+ // asterix-om, and the optimizer side that packs this tuple uses those types.
+ private static final byte AINT8_TYPE_TAG = 1;
+ private static final byte AINT16_TYPE_TAG = 2;
+ private static final byte AINT32_TYPE_TAG = 3;
+ private static final byte AINT64_TYPE_TAG = 4;
+
@Override
protected void resetSearchPredicate(int tupleIndex) {
// Update queryParamsTuple to point to current input tuple
@@ -196,46 +207,71 @@ public class VTreeSearchOperatorNodePushable extends IndexSearchOperatorNodePush
vectorPred.setQueryTuple(queryParamsTuple);
vectorPred.setQueryFieldIndex(QP_FIELD_QUERY_VECTOR);
- // Extract K value if available (+1 to skip the ADM type tag)
- if (queryFields.length > QP_FIELD_K) {
- int k = IntegerPointable.getInteger(queryParamsTuple.getFieldData(QP_FIELD_K),
- queryParamsTuple.getFieldStart(QP_FIELD_K) + 1);
- vectorPred.setK(k);
- }
-
- // Extract min_probe_fraction (double, +1 to skip type tag).
- // Fraction of leaf clusters to probe (0.0-1.0). 0 means use default (0.1).
- if (queryFields.length > QP_FIELD_MIN_PROBE_FRACTION) {
- double minProbeFraction =
- DoublePointable.getDouble(queryParamsTuple.getFieldData(QP_FIELD_MIN_PROBE_FRACTION),
- queryParamsTuple.getFieldStart(QP_FIELD_MIN_PROBE_FRACTION) + 1);
- if (minProbeFraction > 0.0) {
- vectorPred.setMinProbeFraction(minProbeFraction);
- }
- }
-
- // Extract k_multiplier (int, +1 to skip type tag)
- if (queryFields.length > QP_FIELD_K_MULTIPLIER) {
- int queryKMult = IntegerPointable.getInteger(queryParamsTuple.getFieldData(QP_FIELD_K_MULTIPLIER),
- queryParamsTuple.getFieldStart(QP_FIELD_K_MULTIPLIER) + 1);
- vectorPred.setKMultiplier(Math.max(1, queryKMult));
- }
-
- // Set tuple filter for INCLUDE field predicates (e.g., year > 2000)
- // This filter is applied at cursor level for proper K counting
- if (tupleFilter != null) {
- vectorPred.setTupleFilter(tupleFilter);
- }
-
- // Session config compiler.vector.kmultiplier overrides query arg if set (kMultiplier > 1 from constructor)
- if (kMultiplier > 1) {
- vectorPred.setKMultiplier(kMultiplier);
- }
+ // EVERY slot below is set unconditionally, on every input row. The predicate instance is reused
+ // across rows, so a slot left untouched silently inherits the previous row's value — which would
+ // turn "0 means use the default" into "0 means whatever the previous row used".
+
+ // K (+1 to skip the ADM type tag). 0 when the plan did not supply the slot.
+ vectorPred.setK(queryFields.length > QP_FIELD_K ? readIntegerQueryParam(QP_FIELD_K) : 0);
+
+ // min_probe_fraction (double, +1 to skip type tag), a fraction in (0, 1] of the epsilon-filtered
+ // candidate clusters. setMinProbeFraction() maps <= 0 to its own default, so pass 0 when absent.
+ vectorPred.setMinProbeFraction(queryFields.length > QP_FIELD_MIN_PROBE_FRACTION
+ ? DoublePointable.getDouble(queryParamsTuple.getFieldData(QP_FIELD_MIN_PROBE_FRACTION),
+ queryParamsTuple.getFieldStart(QP_FIELD_MIN_PROBE_FRACTION) + 1)
+ : 0.0);
+
+ // k_multiplier: the per-query value, unless the session config compiler.vector.kmultiplier is set
+ // (> 1), which currently wins.
+ int queryKMultiplier = queryFields.length > QP_FIELD_K_MULTIPLIER
+ ? Math.max(1, readIntegerQueryParam(QP_FIELD_K_MULTIPLIER)) : 1;
+ vectorPred.setKMultiplier(kMultiplier > 1 ? kMultiplier : queryKMultiplier);
+
+ // Tuple filter for INCLUDE field predicates (e.g. year > 2000), applied at cursor level so that K
+ // counts only passing tuples. null when this search has no pushed filter.
+ vectorPred.setTupleFilter(tupleFilter);
vectorPred.setEpsilon(indexEpsilon);
}
}
+ /**
+ * Read an integer query parameter, honouring its ADM type tag.
+ * <p>
+ * A fixed 4-byte {@code IntegerPointable} read is not safe on its own: a SQL++ integer literal is an
+ * {@code AInt32} today ({@code LIMIT 3}), but a literal that does not fit in 32 bits, and any folded or
+ * parameterised expression that types as BIGINT, arrives as an 8-byte {@code AInt64} — whose first four
+ * big-endian bytes are 0 for every small value. That produced a silently wrong {@code k} (0) rather than
+ * an error, so switch on the tag instead of assuming the width.
+ */
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.GENERATED)
+ private int readIntegerQueryParam(int fieldIndex) {
+ byte[] data = queryParamsTuple.getFieldData(fieldIndex);
+ int tagOffset = queryParamsTuple.getFieldStart(fieldIndex);
+ int valueOffset = tagOffset + 1; // skip the ADM type tag
+ byte typeTag = data[tagOffset];
+ long value;
+ switch (typeTag) {
+ case AINT8_TYPE_TAG:
+ value = data[valueOffset];
+ break;
+ case AINT16_TYPE_TAG:
+ value = ShortPointable.getShort(data, valueOffset);
+ break;
+ case AINT32_TYPE_TAG:
+ value = IntegerPointable.getInteger(data, valueOffset);
+ break;
+ case AINT64_TYPE_TAG:
+ value = LongPointable.getLong(data, valueOffset);
+ break;
+ default:
+ throw new IllegalStateException("Unexpected type tag " + typeTag + " for integer query parameter at "
+ + "position " + fieldIndex + " of the vector-search query-parameters tuple");
+ }
+ // Clamp rather than overflow: k / k_multiplier are counts used to size buffers.
+ return (int) Math.max(Integer.MIN_VALUE, Math.min(Integer.MAX_VALUE, value));
+ }
+
/**
* Serialized ADOUBLE type tag (value 12), matching {@code ATypeTag.DOUBLE.serialize()} in
* {@code asterix-om}. Hardcoded here because Hyracks cannot depend on asterix-om; the Asterix
diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/LSMVTree.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/LSMVTree.java
index 4abf63d3ac..ac55787f31 100644
--- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/LSMVTree.java
+++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/LSMVTree.java
@@ -682,16 +682,6 @@ public class LSMVTree extends AbstractLSMIndex implements ITreeIndex {
super.destroy();
}
- @Override
- public synchronized void activate() throws HyracksDataException {
- if (isActive) {
- throw HyracksDataException.create(ErrorCode.CANNOT_ACTIVATE_ACTIVE_INDEX);
- }
- loadDiskComponents();
- completeActivation();
- isActive = true;
- }
-
private void loadStaticStructure() throws HyracksDataException {
LSMVTreeFileManager vTreeFileManager = (LSMVTreeFileManager) fileManager;
LSMVTreeComponentFileReferences ssFileRef = vTreeFileManager.getStaticStructureFileReference();
@@ -703,16 +693,17 @@ public class LSMVTree extends AbstractLSMIndex implements ITreeIndex {
setStaticStructure((LSMVTreeDiskComponent) ssComponent);
}
- private void loadDiskComponents() throws HyracksDataException {
- diskComponents.clear();
- List<LSMComponentFileReferences> validFileReferences = fileManager.cleanupAndGetValidFiles();
- for (LSMComponentFileReferences lsmComponentFileReferences : validFileReferences) {
- ILSMDiskComponent component =
- createDiskComponent(componentFactory, lsmComponentFileReferences.getInsertIndexFileReference(),
- lsmComponentFileReferences.getDeleteIndexFileReference(),
- lsmComponentFileReferences.getBloomFilterFileReference(), false);
- diskComponents.add(component);
- }
+ /**
+ * Load the data components exactly as the base class does, then attach the shared static structure —
+ * which is required to navigate any of them.
+ * <p>
+ * The base implementation is sufficient for the data components: this index's file manager leaves the
+ * bloom-filter slot empty (a VTree component has no bloom filter) and carries the shared static structure
+ * in {@link LSMVTreeComponentFileReferences}' own slot, which {@link #loadStaticStructure()} reads.
+ */
+ @Override
+ protected void loadDiskComponents() throws HyracksDataException {
+ super.loadDiskComponents();
loadStaticStructure();
}
}
diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/LSMVTreeFileManager.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/LSMVTreeFileManager.java
index fc8135ace6..9e0831c4f6 100644
--- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/LSMVTreeFileManager.java
+++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/LSMVTreeFileManager.java
@@ -72,8 +72,12 @@ public class LSMVTreeFileManager extends AbstractLSMIndexFileManager {
@Override
public LSMComponentFileReferences getRelMergeFileReference(String firstFileName, String lastFileName) {
final String baseName = IndexComponentFileReference.getMergeSequence(firstFileName, lastFileName);
- return new LSMComponentFileReferences(baseDir.getChild(baseName + DELIMITER + VCTREE_SUFFIX), null,
- baseDir.getChild(baseName + DELIMITER + STATIC_STRUCTURE_SUFFIX));
+ // The static structure is shared by every component and is never rewritten by a merge, so the merged
+ // component references the same shared file the flush path references — there is no per-merge
+ // ".staticstructure" file, and nothing ever creates one. It goes in the dedicated static-structure
+ // slot, not the bloom-filter slot.
+ return new LSMVTreeComponentFileReferences(baseDir.getChild(baseName + DELIMITER + VCTREE_SUFFIX), null, null,
+ baseDir.getChild(STATIC_STRUCTURE_SUFFIX));
}
@Override
@@ -85,7 +89,9 @@ public class LSMVTreeFileManager extends AbstractLSMIndexFileManager {
collectVTreeFiles(allVTreeFiles);
// The single shared static structure (one per index) is required to read any component; without a
- // valid one no VTree component can be opened, so drop everything.
+ // valid one no VTree component can be opened, so drop everything. Note that this branch DELETES every
+ // data component, so it must only be taken when the static structure is *known* to be absent — an
+ // I/O error while probing is propagated by validateStaticStructureFile() instead (see there).
FileReference staticStructureFile = baseDir.getChild(STATIC_STRUCTURE_SUFFIX);
if (!validateStaticStructureFile(staticStructureFile)) {
LOGGER.log(Level.TRACE, "Invalid or missing shared .staticstructure file: {}",
@@ -130,7 +136,10 @@ public class LSMVTreeFileManager extends AbstractLSMIndexFileManager {
for (IndexComponentFileReference vTreeFile : survivors) {
LOGGER.log(Level.TRACE, "Valid VTree component: {} (using shared .staticstructure)",
vTreeFile.getSequence());
- validFiles.add(new LSMComponentFileReferences(vTreeFile.getFileRef(), null, staticStructureFile));
+ // Shared static structure goes in its own slot (LSMVTreeComponentFileReferences), not the
+ // bloom-filter slot — a VTree component has no bloom filter.
+ validFiles
+ .add(new LSMVTreeComponentFileReferences(vTreeFile.getFileRef(), null, null, staticStructureFile));
}
return validFiles;
}
@@ -159,41 +168,37 @@ public class LSMVTreeFileManager extends AbstractLSMIndexFileManager {
/**
* Lists all VTree component files in the base directory and appends their
* IndexComponentFileReference forms to the supplied list.
+ * <p>
+ * Goes through {@link IIOManager} rather than {@code baseDir.getFile().list(...)}: on cloud deployments a
+ * component that exists for the resource need not have a local file, so a direct {@code java.io.File}
+ * listing under-reports (and would silently drop valid components here).
*/
- private void collectVTreeFiles(List<IndexComponentFileReference> files) {
- String[] fileNames = baseDir.getFile().list(vTreeFilter);
- if (fileNames == null) {
- return;
- }
- for (String fileName : fileNames) {
- FileReference fileRef = baseDir.getChild(fileName);
+ private void collectVTreeFiles(List<IndexComponentFileReference> files) throws HyracksDataException {
+ for (FileReference fileRef : ioManager.list(baseDir, vTreeFilter)) {
files.add(IndexComponentFileReference.of(fileRef));
}
}
/**
* Validates that a .staticstructure file exists and is valid.
+ * <p>
+ * Returns {@code false} only for a file that is <em>definitely</em> absent. An I/O failure while probing
+ * ("could not determine") is propagated, deliberately: {@link #cleanupAndGetValidFiles()} reads
+ * {@code false} as "this index has no static structure" and deletes every data component, so swallowing a
+ * transient failure here (a cloud hiccup, EMFILE, a slow mount) would destroy an otherwise healthy index
+ * instead of failing activation.
*
* @param staticStructureFile The .staticstructure file to validate
- * @return true if the file exists and is valid, false otherwise
+ * @return true if the file exists and is valid, false if it is known not to exist
+ * @throws HyracksDataException if existence could not be determined
*/
- private boolean validateStaticStructureFile(FileReference staticStructureFile) {
- try {
- // Check if file exists
- if (!ioManager.exists(staticStructureFile)) {
- LOGGER.log(Level.TRACE, "Static structure file does not exist: {}",
- staticStructureFile.getAbsolutePath());
- return false;
- }
-
- LOGGER.log(Level.TRACE, "Static structure file is valid: {}", staticStructureFile.getAbsolutePath());
- return true;
-
- } catch (Exception e) {
- LOGGER.log(Level.TRACE, "Error validating static structure file {}: {}",
- staticStructureFile.getAbsolutePath(), e.getMessage());
+ private boolean validateStaticStructureFile(FileReference staticStructureFile) throws HyracksDataException {
+ if (!ioManager.exists(staticStructureFile)) {
+ LOGGER.log(Level.TRACE, "Static structure file does not exist: {}", staticStructureFile.getAbsolutePath());
return false;
}
+ LOGGER.log(Level.TRACE, "Static structure file is valid: {}", staticStructureFile.getAbsolutePath());
+ return true;
}
/**
diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/LSMVTreeSearchCursor.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/LSMVTreeSearchCursor.java
index 74c1f8422b..77598be05c 100644
--- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/LSMVTreeSearchCursor.java
+++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/LSMVTreeSearchCursor.java
@@ -33,6 +33,7 @@ import org.apache.hyracks.storage.am.lsm.common.api.ILSMComponent.LSMComponentTy
import org.apache.hyracks.storage.am.lsm.common.api.ILSMIndexOperationContext;
import org.apache.hyracks.storage.am.lsm.common.api.ILSMTreeTupleReference;
import org.apache.hyracks.storage.am.lsm.common.impls.LSMIndexSearchCursor;
+import org.apache.hyracks.storage.am.lsm.vector.utils.LSMVTreeUtils;
import org.apache.hyracks.storage.am.vector.impls.ClusterSearchResult;
import org.apache.hyracks.storage.am.vector.impls.VTree;
import org.apache.hyracks.storage.am.vector.impls.VTree.VTreeAccessor;
@@ -104,13 +105,18 @@ public class LSMVTreeSearchCursor extends LSMIndexSearchCursor {
private boolean[] clusterExhausted; // Whether each component exhausted its current cluster
private boolean stopAdvancing; // Flag to stop advancing after K reached
- // Debug counters to track reconciliation
+ // Progress counters that feed cluster-advancement / early-termination decisions. These are load-bearing,
+ // not diagnostics: both are maintained in doNext() (one tuple produced == one doNext() call), never in
+ // doGetTuple(), which a consumer may call any number of times per tuple.
+ private int tuplesProduced; // Tuples handed to the consumer so far (K progress)
+ private int validTuplesFromCurrentCluster; // Tuples produced from the current cluster (empty-cluster nprobe)
+
+ // Debug counters to track reconciliation (trace logging only — see doClose())
private int totalTuplesPopped; // Total tuples popped from priority queue (including cancelled)
private int antimatterTuplesDetected; // Antimatter tuples detected
private int cancellationsMade; // Matter tuples cancelled by antimatter
private int getTupleCallCount; // Track how many times doGetTuple() is called
private int tuplesFilteredOut; // Tuples that failed INCLUDE field filter
- private int validTuplesFromCurrentCluster; // Valid tuples output from current cluster (for empty-cluster nprobe)
// Tuple filter for INCLUDE field predicates (e.g., year > 2000)
// When set, only tuples passing this filter are returned and counted toward K
@@ -189,10 +195,12 @@ public class LSMVTreeSearchCursor extends LSMIndexSearchCursor {
this.cancellationsMade = 0;
this.getTupleCallCount = 0;
this.tuplesFilteredOut = 0;
+ this.tuplesProduced = 0;
this.validTuplesFromCurrentCluster = 0;
// Set up comparator and operational components
cmp = lsmInitialState.getOriginalKeyComparator();
+ LSMVTreeUtils.validateKeyComparators(cmp, pkStartField, numPrimaryKeyFields);
operationalComponents = lsmInitialState.getOperationalComponents();
lsmHarness = lsmInitialState.getLSMHarness();
@@ -424,11 +432,11 @@ public class LSMVTreeSearchCursor extends LSMIndexSearchCursor {
LOGGER.trace(
"Search summary: mode={}, K={}, nprobe={}, levelWiseClusters={}, levelWiseComplete={},"
+ " minClustersProbed={}, tuplesProcessed={}, antimatter={}, cancellations={},"
- + " filteredOut={}, getTupleCalls={}",
+ + " filteredOut={}, tuplesProduced={}, getTupleCalls={}",
fullScanMode ? "MERGE" : "QUERY", K, nprobe,
clusterStrategy != null ? clusterStrategy.getLevelWiseClusterCount() : 0,
clusterStrategy != null && clusterStrategy.isLevelWisePhaseComplete(), getMinClustersProbed(),
- totalTuplesPopped, antimatterTuplesDetected, cancellationsMade, tuplesFilteredOut,
+ totalTuplesPopped, antimatterTuplesDetected, cancellationsMade, tuplesFilteredOut, tuplesProduced,
getTupleCallCount);
}
super.doClose();
@@ -450,6 +458,15 @@ public class LSMVTreeSearchCursor extends LSMIndexSearchCursor {
// Track total tuples popped (including those that may be cancelled)
totalTuplesPopped++;
+
+ // Count the tuple as produced HERE, not in doGetTuple(): getTuple() is not contractually called
+ // exactly once per tuple, so counting there would inflate K-progress for any consumer that reads the
+ // same tuple twice and trigger early termination sooner — silently reducing recall. Both counters
+ // below feed cluster-advancement decisions (see pushIntoQueueAndAdvanceClusterIfNeeded).
+ if (outputElement != null) {
+ tuplesProduced++;
+ validTuplesFromCurrentCluster++;
+ }
}
@Override
@@ -747,9 +764,6 @@ public class LSMVTreeSearchCursor extends LSMIndexSearchCursor {
@Override
public ITupleReference doGetTuple() {
getTupleCallCount++;
- if (outputElement != null) {
- validTuplesFromCurrentCluster++;
- }
return outputElement != null ? outputElement.getTuple() : null;
}
@@ -807,11 +821,11 @@ public class LSMVTreeSearchCursor extends LSMIndexSearchCursor {
@Override
protected boolean isDeleted(PriorityQueueElement element) throws HyracksDataException {
- // Check if tuple has antimatter bit set (indicates deleted record)
+ // Check if tuple has antimatter bit set (indicates deleted record).
// During merge with full-scan mode, tuples may be rebuilt as ArrayTupleReference
- // which doesn't have antimatter bit - treat those as matter tuples
+ // which doesn't have antimatter bit - treat those as matter tuples.
ITupleReference tuple = element.getTuple();
- return ((ILSMTreeTupleReference) tuple).isAntimatter();
+ return tuple instanceof ILSMTreeTupleReference && ((ILSMTreeTupleReference) tuple).isAntimatter();
}
/**
@@ -913,7 +927,7 @@ public class LSMVTreeSearchCursor extends LSMIndexSearchCursor {
// Calculate minimum clusters explored across all components
int minClustersExplored = getMinClustersProbed();
- int resultsCollected = getTupleCallCount;
+ int resultsCollected = tuplesProduced;
if (clusterStrategy.shouldStopAdvancing(minClustersExplored, resultsCollected)) {
// We have enough results AND probed enough clusters - stop advancing
diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/LSMVTreeTopKSearchCursor.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/LSMVTreeTopKSearchCursor.java
index ae1af91c32..1a51cc7e0e 100644
--- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/LSMVTreeTopKSearchCursor.java
+++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/LSMVTreeTopKSearchCursor.java
@@ -27,6 +27,7 @@ import java.util.PriorityQueue;
import java.util.Set;
import org.apache.hyracks.api.context.IHyracksTaskContext;
+import org.apache.hyracks.api.exceptions.ErrorCode;
import org.apache.hyracks.api.exceptions.HyracksDataException;
import org.apache.hyracks.api.util.CleanupUtils;
import org.apache.hyracks.api.util.HyracksConstants;
@@ -37,6 +38,7 @@ import org.apache.hyracks.storage.am.common.tuples.ReferenceFrameTupleReference;
import org.apache.hyracks.storage.am.lsm.common.api.ILSMComponent;
import org.apache.hyracks.storage.am.lsm.common.api.ILSMIndexOperationContext;
import org.apache.hyracks.storage.am.lsm.common.api.ILSMTreeTupleReference;
+import org.apache.hyracks.storage.am.lsm.vector.utils.LSMVTreeUtils;
import org.apache.hyracks.storage.am.vector.api.IVTreeBinaryAccessor;
import org.apache.hyracks.storage.am.vector.api.IVTreeBinaryAccessorFactory;
import org.apache.hyracks.storage.am.vector.api.IVTreeDistanceFunction;
@@ -181,6 +183,18 @@ public class LSMVTreeTopKSearchCursor extends EnforcedIndexCursor implements IVe
this.pkStartField = dataAccessor.pkStartField();
this.numPrimaryKeyFields = ((LSMVTree) opCtx.getIndex()).getNumPrimaryKeyFields();
+ // This cursor is quantized-only: dataAccessor is fixed to the quantized layout, so it reads field 3
+ // as the quantized embedding and locates the PKs at pkStartField = 4. It is selected purely by the
+ // USE_TOPK_SEARCH access parameter, which is independent of the index's quantization — so check the
+ // assumption instead of inheriting it. On a non-quantized index field 3 is a PK/INCLUDE field and the
+ // whole read would be silent garbage.
+ if (!((LSMVTree) opCtx.getIndex()).isQuantized()) {
+ throw HyracksDataException.create(ErrorCode.ILLEGAL_STATE,
+ "LSMVTreeTopKSearchCursor requires a quantized VTree index (USE_TOPK_SEARCH was requested for a "
+ + "non-quantized index)");
+ }
+ LSMVTreeUtils.validateKeyComparators(cmp, pkStartField, numPrimaryKeyFields);
+
// Extract tuple filter from search predicate for INCLUDE field predicates
this.tupleFilter = vectorPred.getTupleFilter();
if (this.tupleFilter != null) {
@@ -264,6 +278,15 @@ public class LSMVTreeTopKSearchCursor extends EnforcedIndexCursor implements IVe
.create(new IllegalArgumentException("Query vector must be provided for naive blocked search"));
}
+ // computeApproximateDistance() dequantizes every candidate with these two, and they arrive
+ // independently of the USE_TOPK_SEARCH flag that selected this cursor (from a quantizer factory
+ // or instance in the index access parameters). Fail here rather than NPE per candidate.
+ if (this.quantizer == null || this.quantizedQueryVector == null) {
+ throw HyracksDataException.create(ErrorCode.ILLEGAL_STATE,
+ "LSMVTreeTopKSearchCursor requires a quantizer and a quantized query vector; none was supplied "
+ + "through the index access parameters");
+ }
+
// Initialize strategy with first component's tree (candidateLimit so we collect 2*K for reranking)
ILSMComponent firstComponent = operationalComponents.get(0);
VTree vTree = (VTree) firstComponent.getIndex();
diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/NprobeClusterSelectionStrategy.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/NprobeClusterSelectionStrategy.java
index db252f90f9..783025d6dc 100644
--- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/NprobeClusterSelectionStrategy.java
+++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/impls/NprobeClusterSelectionStrategy.java
@@ -98,12 +98,14 @@ public class NprobeClusterSelectionStrategy implements IClusterSelectionStrategy
vTree.getInteriorFrameFactory(), vTree.getLeafFrameFactory(), queryVector, distFunc, epsilon,
quantizedQueryVector, quantizer);
- // Compute nprobe from minProbeFraction * totalLeafClusters
- int totalLeafClusters = globalLevelWiseClusters != null ? globalLevelWiseClusters.size() : 1;
- // Always probe at least one cluster: for small indexes floor(total*fraction) can round to
- // 0, which would make the search entirely K-gated. The max(1,...) clamp is intentional and
- // makes nprobe non-monotonic vs the configured fraction for tiny datasets.
- this.nprobe = Math.max(1, (int) Math.floor(totalLeafClusters * minProbeFraction));
+ // nprobe = minProbeFraction * (number of candidate clusters for THIS query), i.e. a fraction
+ // of the epsilon-filtered level-wise list — NOT a fraction of the index's leaf clusters. The
+ // effective probe count therefore also moves with epsilon and with the query vector.
+ int candidateClusterCount = globalLevelWiseClusters != null ? globalLevelWiseClusters.size() : 1;
+ // Always probe at least one cluster: for small candidate lists floor(count*fraction) can
+ // round to 0, which would make the search entirely K-gated. The max(1,...) clamp is
+ // intentional and makes nprobe non-monotonic vs the configured fraction for tiny datasets.
+ this.nprobe = Math.max(1, (int) Math.floor(candidateClusterCount * minProbeFraction));
// Mark first cluster as visited and start getNextCluster() from index 1
// The cursor handles index 0 separately via getFirstCluster()
@@ -113,7 +115,7 @@ public class NprobeClusterSelectionStrategy implements IClusterSelectionStrategy
}
LOGGER.trace("Computed {} level-wise clusters with epsilon={}, minProbeFraction={}, nprobe={}",
- totalLeafClusters, epsilon, minProbeFraction, nprobe);
+ candidateClusterCount, epsilon, minProbeFraction, nprobe);
} catch (HyracksDataException e) {
// Level-wise probing is an optimization over the DFS fallback; on a storage error we can
// still answer via DFS, but that reduces recall, so surface it at WARN with the cause
@@ -124,6 +126,15 @@ public class NprobeClusterSelectionStrategy implements IClusterSelectionStrategy
+ "selection (recall may be reduced)", e);
globalLevelWiseClusters = null;
}
+ } else if (epsilon <= 0.0 && queryVector != null && vTree != null) {
+ // epsilon == 0 is accepted by DDL/query validation but disables the level-wise phase entirely:
+ // globalLevelWiseClusters stays null, nprobe stays 1, getFirstCluster() returns null, and the
+ // search degrades to a greedy DFS from a single cluster. That is a large, silent recall loss, so
+ // say so rather than answering quietly.
+ LOGGER.warn(
+ "epsilon={} disables level-wise cluster selection; falling back to greedy DFS from a "
+ + "single cluster (recall will be reduced). Use a positive epsilon for ANN search.",
+ epsilon);
}
}
diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/utils/LSMVTreeUtils.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/utils/LSMVTreeUtils.java
index 88b9046f5e..a2be51eb6e 100644
--- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/utils/LSMVTreeUtils.java
+++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/utils/LSMVTreeUtils.java
@@ -24,6 +24,7 @@ import java.util.List;
import org.apache.hyracks.api.dataflow.value.IBinaryComparatorFactory;
import org.apache.hyracks.api.dataflow.value.ITypeTraits;
import org.apache.hyracks.api.dataflow.value.RecordDescriptor;
+import org.apache.hyracks.api.exceptions.ErrorCode;
import org.apache.hyracks.api.exceptions.HyracksDataException;
import org.apache.hyracks.api.io.FileReference;
import org.apache.hyracks.api.io.IIOManager;
@@ -56,7 +57,9 @@ import org.apache.hyracks.storage.am.vector.frames.VTreeInteriorFrameFactory;
import org.apache.hyracks.storage.am.vector.frames.VTreeLeafFrameFactory;
import org.apache.hyracks.storage.am.vector.frames.VTreeMetadataFrameFactory;
import org.apache.hyracks.storage.am.vector.utils.CrossPollinationConfig;
+import org.apache.hyracks.storage.common.MultiComparator;
import org.apache.hyracks.storage.common.buffercache.IBufferCache;
+import org.apache.hyracks.util.annotations.AiProvenance;
/**
* Factory helper that wires the four frame factories, the {@link VTreeFactory}, and the LSM glue
@@ -73,6 +76,31 @@ public final class LSMVTreeUtils {
private LSMVTreeUtils() {
}
+ /**
+ * Assert that the search key comparators cover {@code <distance, PK…>} — i.e. that there is a comparator
+ * for field 0 and for each of the {@code numPrimaryKeyFields} fields starting at {@code pkStartField}.
+ * <p>
+ * Both vector search cursors build their ordering / antimatter-cancellation key from exactly those
+ * positions, and both derive "how many PK fields can I compare" as
+ * {@code min(comparators.length - pkStartField, numPrimaryKeyFields)}. A short comparator array therefore
+ * does not fail — it silently shortens the key, and a zero-length key is worse than an error:
+ * ordering collapses to distance-only (so pairwise antimatter cancellation can fire against an unrelated
+ * same-distance row) and "same primary key" becomes universally true (so group reconciliation keeps only
+ * the newest tuple of an equal-distance group and drops the rest). Production wiring satisfies the
+ * precondition; this turns a future mis-wiring into a hard failure instead of wrong results.
+ */
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.GENERATED)
+ public static void validateKeyComparators(MultiComparator cmp, int pkStartField, int numPrimaryKeyFields)
+ throws HyracksDataException {
+ int available = cmp == null ? 0 : cmp.getComparators().length;
+ if (available < pkStartField + numPrimaryKeyFields) {
+ throw HyracksDataException.create(ErrorCode.ILLEGAL_STATE,
+ "VTree search key comparators too few: " + available + " comparators, but the <distance, PK> key "
+ + "needs " + (pkStartField + numPrimaryKeyFields) + " (pkStartField=" + pkStartField
+ + ", numPrimaryKeyFields=" + numPrimaryKeyFields + ")");
+ }
+ }
+
/**
* Build a configured {@link LSMVTree} from the per-resource configuration. Constructs the four
* frame factories (interior/leaf/metadata pick up their fixed schemas internally; data takes
diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/frames/VTreeMetadataFrame.java b/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/frames/VTreeMetadataFrame.java
index 62e6d98c6d..bdd0f68863 100644
--- a/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/frames/VTreeMetadataFrame.java
+++ b/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/frames/VTreeMetadataFrame.java
@@ -109,6 +109,12 @@ public class VTreeMetadataFrame extends VTreeNSMFrame implements IVTreeMetadataF
* Split a full metadata frame into two halves and insert {@code tuple} into whichever side preserves the
* {@code max_distance}-ascending invariant. Uses a copy-and-reinitialize strategy (rather than in-place
* deletes) to avoid page fragmentation.
+ * <p>
+ * Both halves are re-initialized, which resets their next-page pointers to the end-of-chain sentinel. Re-linking
+ * the two halves into the directory chain — and, crucially, preserving this page's original successor as the
+ * right half's successor — is the caller's responsibility (see
+ * {@code VTree.handleMetadataPageOverflow}); {@link #getNextPage()} must therefore be read <em>before</em>
+ * calling this method.
*/
public void split(IVTreeMetadataFrame rightFrame, ITupleReference tuple) throws HyracksDataException {
int tupleCount = getTupleCount();
@@ -126,8 +132,11 @@ public class VTreeMetadataFrame extends VTreeNSMFrame implements IVTreeMetadataF
rightTuples[i - splitIndex] = TupleUtils.copyTuple(frameTuple);
}
- initBuffer((byte) 0);
- rightFrame.initBuffer((byte) 0);
+ // Preserve this frame's level in both halves rather than hardcoding 0: a directory page is not
+ // necessarily at level 0, and initBuffer() is what stamps the level into the page header.
+ byte level = getLevel();
+ initBuffer(level);
+ rightFrame.initBuffer(level);
for (int i = 0; i < leftTuples.length; i++) {
insert(leftTuples[i], i);
diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTree.java b/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTree.java
index c3d28e1d96..2a779e3294 100644
--- a/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTree.java
+++ b/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTree.java
@@ -287,16 +287,9 @@ public class VTree extends AbstractTreeIndex {
long targetDataPageId = findDataPageInMetadataPage(ctx.getMetadataFrame(), distance, isLastInChain);
if (targetDataPageId != -1) {
- // Found appropriate data page - try to insert
- boolean inserted =
- tryInsertIntoDataPage(targetDataPageId, vector, distance, centroidId, originalTuple, ctx);
-
- if (inserted) {
- return; // Successfully inserted
- }
-
- // If insert failed due to space, we need to handle overflow
- handleDataPageOverflow(currentMetadataPageId, vector, distance, centroidId, originalTuple, ctx);
+ // Found appropriate data page - insert into it. insertIntoDataPage() either inserts
+ // directly, compacts and inserts, or splits the page; it never reports "no room".
+ insertIntoDataPage(targetDataPageId, vector, distance, centroidId, originalTuple, ctx);
return;
}
@@ -370,9 +363,12 @@ public class VTree extends AbstractTreeIndex {
}
/**
- * Try to insert into a specific data page. Returns true if successful, false if page is full.
+ * Insert into a specific data page. The page always absorbs the tuple: with contiguous free space it is
+ * inserted directly, with fragmented free space the page is compacted first, and with no free space the
+ * page is split (the tuple then lands in whichever half covers its distance). Any other space status is a
+ * frame-level invariant violation and is reported as {@code ILLEGAL_STATE}.
*/
- private boolean tryInsertIntoDataPage(long dataPageId, double[] vector, double distance, int centroidId,
+ private void insertIntoDataPage(long dataPageId, double[] vector, double distance, int centroidId,
ITupleReference originalTuple, VTreeOpContext ctx) throws HyracksDataException {
ICachedPage dataPage = bufferCache.pin(BufferedFileHandle.getDiskPageId(getFileId(), (int) dataPageId));
@@ -395,7 +391,7 @@ public class VTree extends AbstractTreeIndex {
switch (spaceStatus) {
case SUFFICIENT_CONTIGUOUS_SPACE:
insertSortedIntoDataPage(dataTuple, distance, dataPageId, originalTuple, ctx);
- return true;
+ return;
case SUFFICIENT_SPACE:
// Fix bug-vtree-delete-frame-corruption: reclaimable space exists but is fragmented
// (FREE_SPACE_OFFSET has been pushed past the slot region by prior inserts whose
@@ -404,12 +400,12 @@ public class VTree extends AbstractTreeIndex {
// BTreeNSMLeafFrame pattern (BTree.java:309-315).
ctx.getDataFrame().compact();
insertSortedIntoDataPage(dataTuple, distance, dataPageId, originalTuple, ctx);
- return true;
+ return;
case INSUFFICIENT_SPACE:
// Handle overflow by splitting the data page (split recomputes the insertion index
// in whichever half the tuple lands, so no position needs to be passed in).
splitDataPageMaintainOrder(ctx.getMetadataPageId(), dataPageId, dataTuple, ctx);
- return true;
+ return;
default:
throw HyracksDataException.create(ErrorCode.ILLEGAL_STATE, "Unexpected FrameOpSpaceStatus "
@@ -428,7 +424,7 @@ public class VTree extends AbstractTreeIndex {
* Insert a data tuple into the currently-latched data frame at its distance-sorted position, fire the
* modification callback, bump the page LSN, and grow the catch-all page's metadata max_distance if this
* distance is a new maximum. Shared by the contiguous-space and post-compaction insert paths in
- * {@link #tryInsertIntoDataPage}.
+ * {@link #insertIntoDataPage}.
*/
private void insertSortedIntoDataPage(ITupleReference dataTuple, double distance, long dataPageId,
ITupleReference originalTuple, VTreeOpContext ctx) throws HyracksDataException {
@@ -651,6 +647,21 @@ public class VTree extends AbstractTreeIndex {
private void handleDataPageOverflow(long metadataPageId, double[] vector, double distance, int centroidId,
ITupleReference originalTuple, VTreeOpContext ctx) throws HyracksDataException {
+ // This method creates the FIRST data page of a directory page and therefore leaves the data-page
+ // chain's next-page pointers alone: with no existing entry there is no predecessor to link from, and
+ // the search cursor reaches data pages only by starting at directory entry 0 and following the chain.
+ // Its single caller reaches it exactly when the directory page has no entries (a page with entries
+ // always yields either a covering entry or the catch-all last entry). Enforce that here rather than
+ // in prose: if this ever runs against a populated directory the new page would be registered in the
+ // directory but unreachable from the chain, i.e. silently invisible to search.
+ VTreeMetadataFrame directoryFrame = requireLatchedMetadataFrame(metadataPageId, ctx);
+ if (directoryFrame.getTupleCount() != 0) {
+ throw HyracksDataException.create(ErrorCode.ILLEGAL_STATE,
+ "handleDataPageOverflow on a non-empty directory page " + metadataPageId + " ("
+ + directoryFrame.getTupleCount()
+ + " entries): the new data page would not be linked into the data-page chain");
+ }
+
// Use the frame factories and page manager to handle overflow
IVTreeDataFrame dataFrame = (IVTreeDataFrame) ctx.getDataFrameFactory().createFrame();
IPageManager pageManager = ctx.getFreePageManager();
@@ -807,14 +818,23 @@ public class VTree extends AbstractTreeIndex {
rightFrame.setPage(newMetadataPage);
rightFrame.initBuffer((byte) 0);
+ // Capture the split page's successor BEFORE splitting: split() re-initializes both halves, which
+ // resets their next-page pointers to the end-of-chain sentinel. The directory page being split is
+ // not necessarily the last in its chain (this method runs on whichever page the insert walk landed
+ // on), so terminating the right half unconditionally would orphan every page after it — making
+ // their data pages unreachable to both the insert walk and tryPhysicalDelete. Splice the new page
+ // in: left -> new -> original successor, mirroring splitDataPageMaintainOrder().
+ int originalNextPage = ctx.getMetadataFrame().getNextPage();
+
// Split the current metadata page using the correct method from VTreeMetadataFrame
((VTreeMetadataFrame) ctx.getMetadataFrame()).split(rightFrame, newTuple);
// Update the next page pointer in the original metadata page
ctx.getMetadataFrame().setNextPage(newMetadataPageId);
- // Initialize the next page pointer in the new metadata page to end-of-chain.
- rightFrame.setNextPage(VTreeDataTupleAccessor.NO_NEXT_PAGE);
+ // The new (right) page inherits the split page's successor, which is the end-of-chain sentinel
+ // exactly when the split page was itself last in the chain.
+ rightFrame.setNextPage(originalNextPage);
} finally {
if (latched) {
diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeBulkLoader.java b/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeBulkLoader.java
index 2c749d03e7..566cd539b7 100644
--- a/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeBulkLoader.java
+++ b/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeBulkLoader.java
@@ -488,11 +488,11 @@ public class VTreeBulkLoader extends PageWriteFailureCallback implements IIndexB
// Destination page ids are deterministic: sequential from staticBasePageId, in source
// page-id order. This lets both passes below run with a bounded number of pages in
// memory (one source pin plus at most one confiscated destination page at a time).
- int staticBasePageId = freePageManager.takePage(metaFrame);
- // Allocate remaining S-1 pages
- for (int i = 1; i < numStaticPages; i++) {
- freePageManager.takePage(metaFrame);
- }
+ // Reserve a CONTIGUOUS block: takeBlock returns the first of numStaticPages consecutive ids, which is
+ // what makes the (staticBasePageId + i) destination math valid. A per-page takePage() loop is NOT safe
+ // here — on a free-list page manager takePage() can hand back non-contiguous ids (see the same
+ // reasoning in VTreeFlushLoader.copyStaticStructure).
+ int staticBasePageId = freePageManager.takeBlock(metaFrame, numStaticPages);
// Create frames for pointer adjustment
IVTreeInteriorFrame intFrame = (IVTreeInteriorFrame) treeIndex.getInteriorFrameFactory().createFrame();
diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeSearchCursor.java b/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeSearchCursor.java
index 56c9a274f0..e7019dbdd1 100644
--- a/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeSearchCursor.java
+++ b/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeSearchCursor.java
@@ -46,6 +46,7 @@ import org.apache.hyracks.storage.common.ISearchPredicate;
import org.apache.hyracks.storage.common.buffercache.IBufferCache;
import org.apache.hyracks.storage.common.buffercache.ICachedPage;
import org.apache.hyracks.storage.common.file.BufferedFileHandle;
+import org.apache.hyracks.util.annotations.AiProvenance;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -534,9 +535,7 @@ public class VTreeSearchCursor extends EnforcedIndexCursor {
LOGGER.warn("VTreeSearchCursor: -1 directory page sentinel in query mode (fileId={}, rootPageId={}); "
+ "treating cluster as empty", fileId, rootPageId);
}
- this.currentDataPageId = -1;
- this.tupleCount = 0;
- this.currentTupleIndex = 0;
+ markCurrentClusterEmpty();
return;
}
@@ -560,9 +559,7 @@ public class VTreeSearchCursor extends EnforcedIndexCursor {
+ "(fileId={}, rootPageId={}); treating cluster as empty",
directoryPageId, fileId, rootPageId);
}
- this.currentDataPageId = -1;
- this.tupleCount = 0;
- this.currentTupleIndex = 0;
+ markCurrentClusterEmpty();
return;
}
@@ -578,11 +575,10 @@ public class VTreeSearchCursor extends EnforcedIndexCursor {
// Open first data page
if (this.currentDataPageId != -1) {
openDataPage(this.currentDataPageId);
+ this.currentTupleIndex = 0;
} else {
- this.tupleCount = 0;
+ markCurrentClusterEmpty();
}
-
- this.currentTupleIndex = 0;
}
/**
@@ -808,14 +804,34 @@ public class VTreeSearchCursor extends EnforcedIndexCursor {
}
}
+ /**
+ * Position the cursor on an empty cluster: release the page of the cluster being abandoned and drop the
+ * frame that was reading it.
+ * <p>
+ * Clearing {@code dataFrame}/{@code frameTuple} is what makes this safe. Leaving them pointing at the
+ * previous cluster's page lets the next {@code hasNext()} fall into {@link #moveToNextDataPage()}, which
+ * reads {@code dataFrame.getNextPage()} off that stale frame and silently resumes the abandoned cluster's
+ * data-page chain. That is reachable in practice because the LSM layer re-points this cursor mid-cluster
+ * during cross-component cluster synchronization ({@code openClusterByResult}), so a cluster is not
+ * necessarily drained to the end of its chain before the next one is opened.
+ */
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.GENERATED)
+ private void markCurrentClusterEmpty() throws HyracksDataException {
+ closeCurrentPage();
+ this.dataFrame = null;
+ this.frameTuple = null;
+ this.currentDataPageId = -1;
+ this.tupleCount = 0;
+ this.currentTupleIndex = 0;
+ }
+
/**
* Open a specific cluster for scanning.
* Gets metadata page, then opens first data page.
*/
private void openCluster(ClusterSearchResult cluster) throws HyracksDataException {
if (cluster == null) {
- this.currentDataPageId = -1;
- this.tupleCount = 0;
+ markCurrentClusterEmpty();
return;
}
@@ -830,12 +846,11 @@ public class VTreeSearchCursor extends EnforcedIndexCursor {
if (this.currentDataPageId != -1) {
openDataPage(this.currentDataPageId);
+ this.currentTupleIndex = 0;
} else {
// Empty cluster
- this.tupleCount = 0;
+ markCurrentClusterEmpty();
}
-
- this.currentTupleIndex = 0;
}
/**
diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeStaticStructureBuilder.java b/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeStaticStructureBuilder.java
index f52f92adee..9ae0488944 100644
--- a/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeStaticStructureBuilder.java
+++ b/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeStaticStructureBuilder.java
@@ -101,6 +101,9 @@ public class VTreeStaticStructureBuilder extends PageWriteFailureCallback implem
private final Map<Integer, int[]> leafDirectory = new HashMap<>();
// Leaf page ids in ascending order (the order leaves must be re-uploaded in, for the cloud writer).
private final TreeSet<Integer> leafPageIds = new TreeSet<>();
+ // Whether the leaf resolution/upload pass has already run. Guards against a second pass re-uploading
+ // pages the append-only cloud writer has already seen (it requires monotonically increasing offsets).
+ private boolean leafNeighborsResolved;
// Structure shape
private final int numLevels;
@@ -392,6 +395,10 @@ public class VTreeStaticStructureBuilder extends PageWriteFailureCallback implem
* runs the compression prepare step (unlike the previous pin/flush path, which assumed no compression).
*/
private void resolveAndUploadLeafNeighbors() throws HyracksDataException {
+ if (leafNeighborsResolved) {
+ return;
+ }
+ leafNeighborsResolved = true;
for (int leafPageId : leafPageIds) {
long dpid = BufferedFileHandle.getDiskPageId(fileId, leafPageId);
ICachedPage page = bufferCache.confiscateAndLoad(dpid);
@@ -500,8 +507,17 @@ public class VTreeStaticStructureBuilder extends PageWriteFailureCallback implem
// (e.g. caller called end() with a partially-built page), flush it now.
if (currentPage != null) {
LOGGER.log(Level.TRACE, "end(): flushing unfinalized page {}", currentPageId);
+ boolean flushedLeafPage = currentLevel == numLevels - 1;
writePage(currentPage);
currentPage = null;
+ if (flushedLeafPage) {
+ // writePage() published that page LOCAL-ONLY (leaf placement). transitionToNextLevel() —
+ // the only other caller of the resolution pass — never ran for the leaf level in this case
+ // (a single-level structure, or end() on a partially-built structure), so without this the
+ // page would stay local and never be uploaded in cloud deployments. The pass is
+ // idempotent-guarded, so this cannot double-upload leaves that were already resolved.
+ resolveAndUploadLeafNeighbors();
+ }
}
if (hasFailed()) {