You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@asterixdb.apache.org by al...@apache.org on 2026/09/21 20:11:37 UTC
(asterixdb) branch master updated: [ASTERIXDB-3875][COMP] Return INCLUDE columns from the vector search
This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
The following commit(s) were added to refs/heads/master by this push:
new 7a6e9b1f20 [ASTERIXDB-3875][COMP] Return INCLUDE columns from the vector search
7a6e9b1f20 is described below
commit 7a6e9b1f20f34edeff507ab7b8c8c0ac777ded02
Author: Ali Alsuliman <al...@gmail.com>
AuthorDate: Sat Sep 19 14:02:09 2026 -0700
[ASTERIXDB-3875][COMP] Return INCLUDE columns from the vector search
A vector index's INCLUDE columns could only be read by a pushed
predicate, so returning one took the query off the index-only plan and
back to fetching and reranking every candidate from the primary index,
to retrieve a value the secondary tuple already held. The search now
declares all of its INCLUDE columns and emits them, and one mapping
keyed by full field path binds whatever reads them, a predicate or a
projection or both. Paths rather than names, so a nested column binds
exactly as a top-level one does, through its covering prefix when the
column holds a record, and primary keys bind through the same call. The
compiler splits a nested access across ASSIGNs, so a path is resolved
through the plan's bindings, on the gate and on the rewrite alike; read
in place, the outer access has a variable for a base and no path. A
field access binds only when rooted at the data record, never the meta
record, whose fields may share a name with a column.
Index selection asks that binding whether the predicate can be
evaluated inside the search, rather than matching field names on its
own. A predicate left in a SELECT above the search filters a candidate
set the search had already capped at k * k_multiplier and chosen
without regard to it, which answers "the passing rows among the k
nearest" instead of "the k nearest passing rows" and can return nothing
where passing rows exist. So no vector index is offered for a
non-functional predicate, which cannot be evaluated per candidate, nor
for a subtree holding more than one SELECT, since the search takes one
condition. An index that is offered is one the pushdown will bind.
The index-only branch splices the search into the plan before retyping
the operators above it: rebound to the search's variables, they have no
producer until then, and a function over one -- year + 1, upper(title),
a field read off an INCLUDE record -- has no input type. The physical
pushdown binds both plan shapes through the same routine, whose
inlining recovers a subexpression moved into an ASSIGN since. It walks
down from a SELECT only through the operators the two plan shapes put
between a WHERE and the search -- a LIMIT ends the walk, a predicate
above one filters a different set of rows -- and roots a field access
only at this dataset's own primary lookup. A SELECT in that pipeline it
cannot bind, or a second one reaching a search that already holds a
condition, fails the compilation: index selection admits only a WHERE
the search can evaluate, so either is a plan reshaped into the shape it
refused. A predicate proven true and removed on the way needs nothing.
Job generation likewise rejects a condition on an index without INCLUDE
columns rather than drop it.
Ext-ref: MB-74040
Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <no...@anthropic.com>
Change-Id: Ibc1a89977219a539203785f778976006912fe4af
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21848
Reviewed-by: Ali Alsuliman <al...@gmail.com>
Reviewed-by: Hongyu Shi <le...@gmail.com>
Reviewed-by: Murtadha Hubail <mh...@apache.org>
Tested-by: Ali Alsuliman <al...@gmail.com>
Integration-Tests: Jenkins <je...@fulliautomatix.ics.uci.edu>
---
.../operators/physical/VectorSearchPOperator.java | 75 ++--
.../rules/PushFilterIntoVectorSearchRule.java | 178 +++++----
.../rules/VectorIncludeFilterPushdown.java | 384 ++++++++++--------
.../rules/am/IntroduceTopKAccessMethodRule.java | 438 ++++++---------------
.../rules/am/VectorIndexAccessMethod.java | 267 ++++++-------
.../queries_sqlpp/vector/VectorQueries.xml | 10 +
...ndex-vtree-include-filter-multi.15.query.sqlpp} | 12 +-
...ndex-vtree-include-filter-multi.16.plans.sqlpp} | 16 +-
...eate-index-vtree-include-filter.11.plans.sqlpp} | 16 +-
...eate-index-vtree-include-filter.16.query.sqlpp} | 13 +-
...eate-index-vtree-include-filter.17.plans.sqlpp} | 17 +-
...eate-index-vtree-include-filter.18.query.sqlpp} | 13 +-
...eate-index-vtree-include-filter.19.plans.sqlpp} | 17 +-
...reate-index-vtree-include-filter.8.query.sqlpp} | 14 +-
...reate-index-vtree-include-filter.9.plans.sqlpp} | 21 +-
...reate-index-vtree-include-nested.7.plans.sqlpp} | 17 +-
...reate-index-vtree-include-nested.8.plans.sqlpp} | 18 +-
...te-index-vtree-include-open-nested.1.ddl.sqlpp} | 17 +-
...-index-vtree-include-open-nested.2.update.sqlpp | 52 +++
...te-index-vtree-include-open-nested.3.ddl.sqlpp} | 10 +-
...te-index-vtree-include-open-nested.4.ddl.sqlpp} | 12 +-
...-index-vtree-include-open-nested.5.query.sqlpp} | 10 +-
...-index-vtree-include-open-nested.6.plans.sqlpp} | 16 +-
...-index-vtree-include-open-nested.7.query.sqlpp} | 11 +-
...-index-vtree-include-open-nested.8.plans.sqlpp} | 18 +-
...e-index-vtree-include-open-nested.99.ddl.sqlpp} | 9 +-
.../create-index-vtree-meta-include.1.ddl.sqlpp | 49 +++
...create-index-vtree-meta-include.2.update.sqlpp} | 10 +-
.../create-index-vtree-meta-include.3.ddl.sqlpp} | 9 +-
.../create-index-vtree-meta-include.4.ddl.sqlpp} | 12 +-
.../create-index-vtree-meta-include.5.query.sqlpp} | 12 +-
.../create-index-vtree-meta-include.6.plans.sqlpp} | 16 +-
.../create-index-vtree-meta-include.7.query.sqlpp} | 12 +-
.../create-index-vtree-meta-include.8.plans.sqlpp} | 17 +-
.../create-index-vtree-meta-include.99.ddl.sqlpp} | 9 +-
.../vtree-scan-choice.6.query.sqlpp | 5 +-
...query.sqlpp => vtree-scan-choice.9.query.sqlpp} | 6 +-
.../create-index-vtree-composite-pk.11.plan | 16 +-
.../create-index-vtree-composite-pk.8.plan | 22 +-
...create-index-vtree-include-filter-multi.13.plan | 2 +-
...create-index-vtree-include-filter-multi.14.plan | 2 +-
.../create-index-vtree-include-filter-multi.15.adm | 3 +
...create-index-vtree-include-filter-multi.16.plan | 32 ++
.../create-index-vtree-include-filter-multi.7.plan | 2 +-
.../create-index-vtree-include-filter.11.plan | 30 ++
.../create-index-vtree-include-filter.16.adm | 3 +
.../create-index-vtree-include-filter.17.plan | 32 ++
.../create-index-vtree-include-filter.18.adm | 3 +
.../create-index-vtree-include-filter.19.plan} | 16 +-
.../create-index-vtree-include-filter.8.adm | 3 +
.../create-index-vtree-include-filter.9.plan} | 10 +-
.../create-index-vtree-include-nested.7.plan} | 16 +-
.../create-index-vtree-include-nested.8.plan} | 2 +-
.../create-index-vtree-include-open-nested.5.adm | 3 +
.../create-index-vtree-include-open-nested.6.plan} | 10 +-
.../create-index-vtree-include-open-nested.7.adm | 3 +
.../create-index-vtree-include-open-nested.8.plan | 32 ++
.../create-index-vtree-meta-include.5.adm | 3 +
.../create-index-vtree-meta-include.6.plan} | 10 +-
.../create-index-vtree-meta-include.7.adm | 3 +
.../create-index-vtree-meta-include.8.plan | 32 ++
.../create-index-vtree-nested.5.plan | 16 +-
.../vtree-scan-choice/vtree-scan-choice.5.regex | 4 +
63 files changed, 1180 insertions(+), 968 deletions(-)
diff --git a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/VectorSearchPOperator.java b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/VectorSearchPOperator.java
index c4d9e3d13f..d433c87785 100644
--- a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/VectorSearchPOperator.java
+++ b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/VectorSearchPOperator.java
@@ -31,7 +31,6 @@ import org.apache.asterix.metadata.declared.MetadataProvider;
import org.apache.asterix.metadata.entities.Dataset;
import org.apache.asterix.metadata.entities.Index;
import org.apache.asterix.om.functions.BuiltinFunctions;
-import org.apache.asterix.om.types.IAType;
import org.apache.asterix.optimizer.rules.VectorIncludeFilterPushdown;
import org.apache.asterix.optimizer.rules.am.VectorJobGenParams;
import org.apache.hyracks.algebricks.common.constraints.AlgebricksPartitionConstraint;
@@ -68,7 +67,7 @@ import org.apache.hyracks.util.annotations.AiProvenance;
*/
public class VectorSearchPOperator extends IndexSearchPOperator {
- private static final int[] NO_INCLUDE_FILTER_FIELDS = new int[0];
+ private static final int[] NO_INCLUDE_FIELDS = new int[0];
public VectorSearchPOperator(IDataSourceIndex<String, DataSourceId> idx, INodeDomain domain,
boolean requiresBroadcast) {
@@ -121,6 +120,7 @@ public class VectorSearchPOperator extends IndexSearchPOperator {
}
@Override
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_FABLE_5_1, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.ASSISTED, notes = "Reject a filter condition on an index without INCLUDE columns instead of dropping it")
public void contributeRuntimeOperator(IHyracksJobBuilder builder, JobGenContext context, ILogicalOperator op,
IOperatorSchema opSchema, IOperatorSchema[] inputSchemas, IOperatorSchema outerPlanSchema)
throws AlgebricksException {
@@ -160,48 +160,36 @@ public class VectorSearchPOperator extends IndexSearchPOperator {
boolean isQuantized = vectorDetails.getVectorParameters().isQuantized();
int numSecondaryKeys = VTreeDataTupleAccessor.getNumSecondaryFields(isQuantized);
- // Create tuple filter factory if selectCondition is present (for INCLUDE field filtering)
- // The opSchema only has [pk] because INCLUDE fields are only used for filtering.
- // Filter variables are mapped directly to physical field indexes via annotation.
+ // The INCLUDE columns are declared on the search whether this query reads them or not, so one
+ // mapping serves the pushed predicate, the projection, and the runtime's output tuple alike.
ITupleFilterFactory tupleFilterFactory = null;
- int[] includeFilterFields = NO_INCLUDE_FILTER_FIELDS;
+ int[] includeFields = NO_INCLUDE_FIELDS;
if (unnestMap instanceof UnnestMapOperator) {
UnnestMapOperator unnestMapOp = (UnnestMapOperator) unnestMap;
- if (unnestMapOp.getSelectCondition() != null) {
- // Get filter variable to physical field index mapping from annotation
- @SuppressWarnings("unchecked")
- Map<LogicalVariable, Integer> filterVarToFieldIndex = (Map<LogicalVariable, Integer>) unnestMapOp
- .getAnnotations().get(VectorIncludeFilterPushdown.VECTOR_FILTER_VAR_MAPPING);
-
- // Get filter variable types from annotation
- @SuppressWarnings("unchecked")
- Map<LogicalVariable, IAType> filterVarTypes = (Map<LogicalVariable, IAType>) unnestMapOp
- .getAnnotations().get(VectorIncludeFilterPushdown.VECTOR_FILTER_VAR_TYPES);
+ VectorIncludeFilterPushdown.IncludeColumns columns =
+ VectorIncludeFilterPushdown.getIncludeColumns(unnestMapOp);
- // Create filter schema with direct mapping for filter-only variables
- // numSecondaryKeys: offset from physical tuple start to PK field
+ if (unnestMapOp.getSelectCondition() != null) {
+ if (columns == null) {
+ // A condition reaches the search only through the binding, which has nothing to bind a
+ // field access to without INCLUDE columns. One here without them was installed by some
+ // other hand, and dropping it would return the rows it was meant to remove.
+ throw new CompilationException(ErrorCode.COMPILATION_ILLEGAL_STATE, unnestMap.getSourceLocation(),
+ "the vector index search of index " + jobGenParams.getIndexName()
+ + " carries a filter condition but the index declares no INCLUDE columns");
+ }
+ // The operator schema carries the INCLUDE columns at their output positions, but the filter
+ // runs against the SECONDARY tuple, so its variables resolve through the mapping instead.
IOperatorSchema filterSchema =
- new VectorIndexFilterSchema(opSchema, filterVarToFieldIndex, numSecondaryKeys);
-
- // Create type environment with filter variable types
- // Pass context so function expressions can use this wrapper for recursive type lookups
+ new VectorIndexFilterSchema(opSchema, columns.varToFieldIndex(), numSecondaryKeys);
IVariableTypeEnvironment filterTypeEnv =
- new VectorIndexFilterTypeEnvironment(typeEnv, filterVarTypes, context);
-
+ new VectorIndexFilterTypeEnvironment(typeEnv, columns.varTypes(), context);
tupleFilterFactory = mp.createTupleFilterFactory(new IOperatorSchema[] { filterSchema }, filterTypeEnv,
unnestMapOp.getSelectCondition().getValue(), context);
+ }
- includeFilterFields = includeFilterFields(unnestMapOp, filterVarToFieldIndex);
- } else if (VectorIncludeFilterPushdown.hasDeclaredFilterVariables(unnestMapOp)) {
- // The index-only plan declares the INCLUDE columns its predicate reads as outputs of this
- // unnest-map while leaving the predicate in a SELECT above, for PushFilterIntoVectorSearchRule
- // to move into the select condition. Reaching job generation with the columns declared but no
- // condition means that never happened -- something relocated or consumed the SELECT. The
- // runtime emits only [pk..., dist], so the declared columns would be read as fields that were
- // never written. Fail here rather than emit that.
- throw new CompilationException(ErrorCode.COMPILATION_ILLEGAL_STATE, unnestMap.getSourceLocation(),
- "the vector index-only plan declared INCLUDE filter columns on the index search of "
- + jobGenParams.getIndexName() + " but no filter condition reached it");
+ if (columns != null) {
+ includeFields = includeFieldIndexes(unnestMapOp, columns.varToFieldIndex());
}
}
@@ -211,7 +199,7 @@ public class VectorSearchPOperator extends IndexSearchPOperator {
Pair<IOperatorDescriptor, AlgebricksPartitionConstraint> vectorSearch =
mp.getVectorSearchRuntime(builder.getJobSpec(), outputVars, opSchema, typeEnv, context,
jobGenParams.getRetainInput(), dataset, jobGenParams.getIndexName(), queryIndexes,
- tupleFilterFactory, includeFilterFields, jobGenParams.isIndexOnly());
+ tupleFilterFactory, includeFields, jobGenParams.isIndexOnly());
IOperatorDescriptor opDesc = vectorSearch.first;
opDesc.setSourceLocation(unnestMap.getSourceLocation());
@@ -224,21 +212,22 @@ public class VectorSearchPOperator extends IndexSearchPOperator {
}
/**
- * The physical field indexes of the INCLUDE columns the pushed filter reads, in output order.
+ * The physical field indexes of the INCLUDE columns this search emits, in output order.
* <p>
- * Those columns are the LAST variables of the unnest-map: both call sites of
- * {@link VectorIncludeFilterPushdown#declareFilterVariables} append them, the index-only one after it has
- * appended the distance. The operator schema, and hence the output record descriptor, is built from that
- * same list, and the runtime writes the primary keys, then the distance, then these columns -- so their
+ * Those columns are the LAST variables of the unnest-map:
+ * {@link VectorIncludeFilterPushdown#declareIncludeColumns} appends them after the primary keys and, on
+ * the index-only plan, after the distance. The operator schema, and hence the output record descriptor,
+ * is built from that same list, and the runtime writes the primary keys, then the distance, then these
+ * columns -- so their
* being trailing and in this order is what makes the two agree. Checked rather than assumed: a tuple
* narrower than the descriptor is emitted with the previous tuple's offsets in its trailing slots, not
* with an error.
*/
@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.GENERATED)
- private static int[] includeFilterFields(UnnestMapOperator unnestMap,
+ private static int[] includeFieldIndexes(UnnestMapOperator unnestMap,
Map<LogicalVariable, Integer> filterVarToFieldIndex) throws CompilationException {
if (filterVarToFieldIndex == null || filterVarToFieldIndex.isEmpty()) {
- return NO_INCLUDE_FILTER_FIELDS;
+ return NO_INCLUDE_FIELDS;
}
List<LogicalVariable> vars = unnestMap.getVariables();
int numFilterVars = filterVarToFieldIndex.size();
diff --git a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushFilterIntoVectorSearchRule.java b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushFilterIntoVectorSearchRule.java
index 0c85eeb852..37fad096fd 100644
--- a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushFilterIntoVectorSearchRule.java
+++ b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/PushFilterIntoVectorSearchRule.java
@@ -21,9 +21,12 @@ package org.apache.asterix.optimizer.rules;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
+import java.util.Objects;
import java.util.Set;
import org.apache.asterix.common.config.DatasetConfig.IndexType;
+import org.apache.asterix.common.exceptions.CompilationException;
+import org.apache.asterix.common.exceptions.ErrorCode;
import org.apache.asterix.metadata.declared.MetadataProvider;
import org.apache.asterix.metadata.entities.Dataset;
import org.apache.asterix.metadata.entities.Index;
@@ -31,8 +34,8 @@ import org.apache.asterix.om.functions.BuiltinFunctions;
import org.apache.asterix.om.types.ARecordType;
import org.apache.asterix.optimizer.rules.am.AccessMethodJobGenParams;
import org.apache.commons.lang3.mutable.Mutable;
-import org.apache.commons.lang3.mutable.MutableObject;
import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException;
+import org.apache.hyracks.algebricks.common.utils.Pair;
import org.apache.hyracks.algebricks.core.algebra.base.ILogicalExpression;
import org.apache.hyracks.algebricks.core.algebra.base.ILogicalOperator;
import org.apache.hyracks.algebricks.core.algebra.base.IOptimizationContext;
@@ -69,8 +72,19 @@ import org.apache.hyracks.util.annotations.AiProvenance;
* selectCondition: (rewritten to use $includeField1, ...)
* </pre>
*
- * New variables are created for the INCLUDE fields produced by VECTOR_INDEX_UNNEST, and
+ * The INCLUDE columns are already declared as outputs of VECTOR_INDEX_UNNEST by the access-method phase;
* field-access expressions in the filter are rewritten to reference those variables.
+ * <p>
+ * The SELECT has to sit within the search's own pipeline: the descent from it stops at a LIMIT, or at
+ * any operator other than the ones the two plan shapes put between a WHERE and the search. A predicate
+ * above a LIMIT filters the rows the LIMIT let through, which is a different query from filtering the
+ * candidates the search ranks, so it must stay where it is.
+ * <p>
+ * A SELECT that does sit in the pipeline is the WHERE index selection admitted, and it was admitted because
+ * the search can evaluate it. One that cannot be bound here, or a second one reaching a search that already
+ * holds a condition, is therefore a plan some rule in between has reshaped into the very thing index selection
+ * refused -- a filter left above a search that has already capped its candidates -- and compilation fails
+ * rather than emit it. A predicate proven true and removed on the way never arrives here and needs nothing.
*/
public class PushFilterIntoVectorSearchRule implements IAlgebraicRewriteRule {
@@ -99,35 +113,30 @@ public class PushFilterIntoVectorSearchRule implements IAlgebraicRewriteRule {
return false;
}
- // Index-only shape: the access-method phase already declared the INCLUDE columns on the unnest-map
- // and rebound the predicate to them, leaving it here as an ordinary SELECT so that the rules in
- // between saw an ordinary plan. Nothing is left to resolve -- just move it into the select
- // condition, which is the one place the runtime reads it from.
- if (VectorIncludeFilterPushdown.hasDeclaredFilterVariables(searchInfo.vectorUnnest())) {
- Set<LogicalVariable> conditionVars = new HashSet<>();
- selectOp.getCondition().getValue().getUsedVariables(conditionVars);
- if (!searchInfo.vectorUnnest().getVariables().containsAll(conditionVars)) {
- return false;
- }
- searchInfo.vectorUnnest().setSelectCondition(new MutableObject<>(selectOp.getCondition().getValue()));
- return dropSelect(opRef, selectOp, searchInfo.vectorUnnest(), op, context);
+ UnnestMapOperator vectorUnnest = searchInfo.vectorUnnest();
+ if (vectorUnnest.getSelectCondition() != null) {
+ throw new CompilationException(ErrorCode.COMPILATION_ILLEGAL_STATE, selectOp.getSourceLocation(),
+ "a second WHERE reached the vector search of index " + searchInfo.indexName()
+ + ", which evaluates one condition");
}
- // The whole decision — which field paths the predicate reads, whether the index's INCLUDE list
- // covers them, and what the rewritten predicate looks like — lives in VectorIncludeFilterPushdown,
- // shared with the index-only gate in IntroduceTopKAccessMethodRule. See that class for why the two
- // must not be separate implementations.
+ // One binding for both plan shapes. On the index-only plan the access-method phase already rebound
+ // the predicate onto the INCLUDE columns, so there is no field access left to rewrite -- but the
+ // rules that ran since may have moved part of it into an ASSIGN (ExtractCommonExpressionsRule does,
+ // for a subexpression the projection shares), and only the binding's inlining puts that back. The
+ // completeness guard then confirms nothing but search outputs remain, on either shape.
VectorIncludeFilterPushdown.IndexContext idx =
new VectorIncludeFilterPushdown.IndexContext(searchInfo.includeFieldNames(), searchInfo.recordType(),
searchInfo.isQuantized(), searchInfo.numPrimaryKeys(), searchInfo.recordVars());
- VectorIncludeFilterPushdown.PushedIncludeFilter pushed = VectorIncludeFilterPushdown
- .analyze(selectOp.getCondition().getValue(), selectOp, idx, context, context::newVar);
- if (pushed == null) {
- return false;
+ ILogicalExpression bound = VectorIncludeFilterPushdown.bindPredicate(selectOp.getCondition().getValue(),
+ selectOp, idx, context, VectorIncludeFilterPushdown.getIncludeColumns(vectorUnnest));
+ if (bound == null) {
+ throw new CompilationException(ErrorCode.COMPILATION_ILLEGAL_STATE, selectOp.getSourceLocation(),
+ "the WHERE above the vector search of index " + searchInfo.indexName()
+ + " reads what the search cannot evaluate, though index selection admits only a WHERE it can");
}
-
- VectorIncludeFilterPushdown.apply(searchInfo.vectorUnnest(), pushed);
- return dropSelect(opRef, selectOp, searchInfo.vectorUnnest(), op, context);
+ VectorIncludeFilterPushdown.apply(vectorUnnest, bound);
+ return dropSelect(opRef, selectOp, vectorUnnest, op, context);
}
/** Remove the SELECT whose predicate now lives in the vector search, and retype what changed. */
@@ -147,70 +156,89 @@ public class PushFilterIntoVectorSearchRule implements IAlgebraicRewriteRule {
* (and PK) variables of THIS search's primary-index lookup, excluding its meta
* record. A field access must be rooted at one of these to be a candidate for
* pushdown.
+ * @param indexName the searched index, for diagnostics
*/
private record VectorSearchInfo(UnnestMapOperator vectorUnnest, List<List<String>> includeFieldNames,
- ARecordType recordType, boolean isQuantized, int numPrimaryKeys, Set<LogicalVariable> recordVars) {
+ ARecordType recordType, boolean isQuantized, int numPrimaryKeys, Set<LogicalVariable> recordVars,
+ String indexName) {
}
/**
- * Finds VECTOR_INDEX_UNNEST below the SELECT operator, skipping intervening ASSIGNs.
+ * Finds the vector index search this SELECT's predicate can be moved into, or {@code null} if there is
+ * none within the search's own pipeline.
*/
private VectorSearchInfo findVectorIndexUnnest(SelectOperator selectOp, IOptimizationContext context)
throws AlgebricksException {
- ILogicalOperator current = selectOp.getInputs().get(0).getValue();
- while (current.getOperatorTag() == LogicalOperatorTag.ASSIGN) {
- current = current.getInputs().get(0).getValue();
- }
- return searchForVectorUnnest(current, context, new ArrayList<>());
+ return searchForVectorUnnest(selectOp.getInputs().get(0).getValue(), context, new ArrayList<>());
}
/**
- * Recursively searches for a VECTOR_INDEX_UNNEST under the given operator, collecting on the way down
- * the non-vector unnest-maps it passes through — the primary-index lookup that materializes the dataset
- * record the filter reads. Their variables become the pushdown bases in {@link #buildSearchInfo}, which
- * is where the dataset is known and the meta record can be told apart from the record.
+ * Walks down from the SELECT to a VECTOR_INDEX_UNNEST, collecting on the way the index-search
+ * unnest-maps it passes through — the primary-index lookup that materializes the dataset record the
+ * filter reads. Their variables become the pushdown bases in {@link #buildSearchInfo}, which is where the
+ * dataset is known and the meta record can be told apart from the record.
+ * <p>
+ * Only the operators the two plan shapes put between a WHERE and the search are walked through, and each
+ * must have a single input. A LIMIT in particular ends the walk: a predicate above it applies to the rows
+ * the LIMIT let through, and moving it into the search would instead choose which candidates the search
+ * ranks — "the k nearest that pass" in place of "those of the k nearest that pass". A join ends it too, so
+ * a predicate over another branch's record can never be resolved against this index's INCLUDE columns.
*/
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_FABLE_5_1, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.REFACTORED, notes = "Stop the descent at a LIMIT and at operators outside the search pipeline")
private VectorSearchInfo searchForVectorUnnest(ILogicalOperator op, IOptimizationContext context,
- List<UnnestMapOperator> recordSources) throws AlgebricksException {
-
- if (op.getOperatorTag() == LogicalOperatorTag.UNNEST_MAP) {
- UnnestMapOperator unnest = (UnnestMapOperator) op;
- ILogicalExpression expr = unnest.getExpressionRef().getValue();
-
- if (expr.getExpressionTag() == LogicalExpressionTag.FUNCTION_CALL) {
- AbstractFunctionCallExpression funcExpr = (AbstractFunctionCallExpression) expr;
-
- if (funcExpr.getFunctionIdentifier().equals(BuiltinFunctions.INDEX_SEARCH)) {
- AccessMethodJobGenParams params = new AccessMethodJobGenParams();
- params.readFromFuncArgs(funcExpr.getArguments());
-
- if (params.getIndexType() == IndexType.VTREE) {
- if (unnest.getSelectCondition() != null) {
- return null;
- }
- return buildSearchInfo(unnest, params, context, recordSources);
- }
+ List<Pair<UnnestMapOperator, AccessMethodJobGenParams>> recordSources) throws AlgebricksException {
+ switch (op.getOperatorTag()) {
+ case UNNEST_MAP: {
+ UnnestMapOperator unnest = (UnnestMapOperator) op;
+ AccessMethodJobGenParams params = indexSearchParams(unnest);
+ if (params == null) {
+ return null;
+ }
+ if (params.getIndexType() == IndexType.VTREE) {
+ return buildSearchInfo(unnest, params, context, recordSources);
}
+ recordSources.add(new Pair<>(unnest, params));
+ break;
}
- recordSources.add(unnest);
+ case ASSIGN:
+ case SELECT:
+ case PROJECT:
+ case ORDER:
+ case DISTINCT:
+ case EXCHANGE:
+ break;
+ default:
+ return null;
}
-
- for (Mutable<ILogicalOperator> inputRef : op.getInputs()) {
- VectorSearchInfo result = searchForVectorUnnest(inputRef.getValue(), context, recordSources);
- if (result != null) {
- return result;
- }
+ if (op.getInputs().size() != 1) {
+ return null;
}
+ return searchForVectorUnnest(op.getInputs().get(0).getValue(), context, recordSources);
+ }
- return null;
+ /** The parameters of the index search {@code unnest} performs, or {@code null} if it is not one. */
+ private static AccessMethodJobGenParams indexSearchParams(UnnestMapOperator unnest) throws AlgebricksException {
+ ILogicalExpression expr = unnest.getExpressionRef().getValue();
+ if (expr.getExpressionTag() != LogicalExpressionTag.FUNCTION_CALL) {
+ return null;
+ }
+ AbstractFunctionCallExpression funcExpr = (AbstractFunctionCallExpression) expr;
+ if (!funcExpr.getFunctionIdentifier().equals(BuiltinFunctions.INDEX_SEARCH)) {
+ return null;
+ }
+ AccessMethodJobGenParams params = new AccessMethodJobGenParams();
+ params.readFromFuncArgs(funcExpr.getArguments());
+ return params;
}
/**
* Builds VectorSearchInfo from the found vector index.
*/
@AiProvenance(agent = AiProvenance.Agent.CLAUDE_FABLE_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.ASSISTED)
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_FABLE_5_1, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.ASSISTED, notes = "Admit only this dataset's primary lookup as a pushdown base")
private VectorSearchInfo buildSearchInfo(UnnestMapOperator unnest, AccessMethodJobGenParams params,
- IOptimizationContext context, List<UnnestMapOperator> recordSources) throws AlgebricksException {
+ IOptimizationContext context, List<Pair<UnnestMapOperator, AccessMethodJobGenParams>> recordSources)
+ throws AlgebricksException {
MetadataProvider mp = (MetadataProvider) context.getMetadataProvider();
@@ -230,17 +258,31 @@ public class PushFilterIntoVectorSearchRule implements IAlgebraicRewriteRule {
ARecordType recordType = (ARecordType) mp.findType(dataset.getItemTypeDatabaseName(),
dataset.getItemTypeDataverseName(), dataset.getItemTypeName());
- // A primary-index lookup on a collection with a meta part produces [pk..., record, meta]. Drop the
+ // Only this dataset's own primary-index lookup materializes the record the INCLUDE paths are resolved
+ // against; a field access on any other lookup's record must be left alone so the completeness guard
+ // declines it. On a collection with a meta part the lookup produces [pk..., record, meta]. Drop the
// meta variable: resolveFieldPath resolves a field access against the dataset's record type, so
// admitting it would let `WHERE meta(m).year > 0` bind to the record's INCLUDE column of the same
// name and filter on the wrong value.
Set<LogicalVariable> recordVars = new HashSet<>();
- for (UnnestMapOperator lookup : recordSources) {
- List<LogicalVariable> vars = lookup.getVariables();
+ for (Pair<UnnestMapOperator, AccessMethodJobGenParams> lookup : recordSources) {
+ if (!isPrimaryLookupOf(lookup.second, dataset)) {
+ continue;
+ }
+ List<LogicalVariable> vars = lookup.first.getVariables();
recordVars.addAll(vars.subList(0, dataset.hasMetaPart() ? vars.size() - 1 : vars.size()));
}
return new VectorSearchInfo(unnest, details.getIncludeFieldNames(), recordType,
- details.getVectorParameters().isQuantized(), dataset.getPrimaryKeys().size(), recordVars);
+ details.getVectorParameters().isQuantized(), dataset.getPrimaryKeys().size(), recordVars,
+ index.getIndexName());
+ }
+
+ /** Whether {@code params} describe a search of {@code dataset}'s primary index, which is named after it. */
+ private static boolean isPrimaryLookupOf(AccessMethodJobGenParams params, Dataset dataset) {
+ return Objects.equals(params.getDatabaseName(), dataset.getDatabaseName())
+ && Objects.equals(params.getDataverseName(), dataset.getDataverseName())
+ && Objects.equals(params.getDatasetName(), dataset.getDatasetName())
+ && Objects.equals(params.getIndexName(), dataset.getDatasetName());
}
}
diff --git a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/VectorIncludeFilterPushdown.java b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/VectorIncludeFilterPushdown.java
index c2e019daf2..24fa258f28 100644
--- a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/VectorIncludeFilterPushdown.java
+++ b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/VectorIncludeFilterPushdown.java
@@ -77,67 +77,168 @@ public final class VectorIncludeFilterPushdown {
}
/**
- * Annotation key for the filter variable to physical field index mapping.
- * Value type: Map<LogicalVariable, Integer>
+ * Annotation key for the INCLUDE columns a vector index search emits.
+ * Value type: {@link IncludeColumns}
*/
- public static final String VECTOR_FILTER_VAR_MAPPING = "VECTOR_FILTER_VAR_MAPPING";
+ public static final String VECTOR_INCLUDE_COLUMNS = "VECTOR_INCLUDE_COLUMNS";
/**
- * Annotation key for the filter variable to type mapping.
- * Value type: Map<LogicalVariable, IAType>
+ * Every INCLUDE column of the searched index, bound to a variable the search emits.
+ * <p>
+ * All of them are declared, whether this query reads them or not, so that one mapping serves every use:
+ * a predicate pushed into the search, a projection returning the column, or neither. Keyed by the full
+ * path, so a nested INCLUDE column ({@code INCLUDE (info.year)}) is addressed exactly like a top-level
+ * one and never confused with a same-named field elsewhere in the record.
+ *
+ * @param pathToVar each column's full path to the variable carrying it, in INCLUDE order; empty when
+ * the index declares the same path twice, which leaves no unambiguous binding
+ * @param varToFieldIndex each variable's physical field index in the secondary tuple
+ * @param varTypes each variable's type, for the filter's type environment
*/
- public static final String VECTOR_FILTER_VAR_TYPES = "VECTOR_FILTER_VAR_TYPES";
+ public record IncludeColumns(Map<List<String>, LogicalVariable> pathToVar,
+ Map<LogicalVariable, Integer> varToFieldIndex, Map<LogicalVariable, IAType> varTypes) {
+ }
/**
- * Register the INCLUDE columns a rewritten predicate reads as variables produced by the vector index
- * unnest-map, and hand {@link org.apache.asterix.algebra.operators.physical.VectorSearchPOperator} their
- * physical field indexes via the annotations above.
+ * Declare every INCLUDE column of the index as an output of the vector index search, appended after the
+ * primary keys and, on the index-only plan, the distance.
* <p>
- * This only declares the columns; the predicate itself stays wherever the caller put it. The index-only
- * plan uses that: it binds its predicate to these variables but leaves it in a {@code SELECT} above the
- * unnest-map, so the predicate remains an ordinary expression over ordinary variables for every rule
- * that runs before {@link PushFilterIntoVectorSearchRule} moves it in.
+ * The runtime emits exactly what is declared here, so the columns are bound once, at the point the
+ * unnest-map is created, rather than discovered per use. A predicate and a projection over the same
+ * column then resolve to the same variable by construction.
*/
- public static void declareFilterVariables(UnnestMapOperator vectorUnnest, PushedIncludeFilter pushed) {
- vectorUnnest.getAnnotations().put(VECTOR_FILTER_VAR_MAPPING, pushed.varToFieldIndex());
- vectorUnnest.getAnnotations().put(VECTOR_FILTER_VAR_TYPES, pushed.varTypes());
-
- for (Map.Entry<LogicalVariable, IAType> entry : pushed.varTypes().entrySet()) {
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.GENERATED)
+ public static IncludeColumns declareIncludeColumns(UnnestMapOperator vectorUnnest, IndexContext idx,
+ Supplier<LogicalVariable> varSupplier) throws AlgebricksException {
+ IncludeColumns columns = buildIncludeColumns(idx, varSupplier);
+ if (columns == null) {
+ return null;
+ }
+ for (Map.Entry<LogicalVariable, IAType> entry : columns.varTypes().entrySet()) {
vectorUnnest.getVariables().add(entry.getKey());
vectorUnnest.getVariableTypes().add(entry.getValue());
}
+ vectorUnnest.getAnnotations().put(VECTOR_INCLUDE_COLUMNS, columns);
+ return columns;
}
/**
- * Declare the INCLUDE columns and move the predicate into the unnest-map's select condition, which is
- * where the runtime reads it from. Only {@link PushFilterIntoVectorSearchRule} calls this, during the
- * physical rewrites: a select condition reads variables the operator produces rather than variables from
- * its input, which is not what an expression on an operator normally means, so it is created as late as
- * possible to keep it out of the way of the rules that walk expressions.
+ * The index's INCLUDE columns bound to variables from {@code varSupplier}, or {@code null} when the
+ * index has none. Touches no operator, so the gate can run the analysis on throwaway variables to reach
+ * a verdict before any of this is spliced into a plan.
*/
- public static void apply(UnnestMapOperator vectorUnnest, PushedIncludeFilter pushed) {
- declareFilterVariables(vectorUnnest, pushed);
- vectorUnnest.setSelectCondition(new MutableObject<>(pushed.condition()));
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.GENERATED)
+ private static IncludeColumns buildIncludeColumns(IndexContext idx, Supplier<LogicalVariable> varSupplier)
+ throws AlgebricksException {
+ List<List<String>> includeFieldNames = idx.includeFieldNames();
+ if (includeFieldNames == null || includeFieldNames.isEmpty()) {
+ return null;
+ }
+ Map<List<String>, LogicalVariable> pathToVar = new LinkedHashMap<>();
+ Map<LogicalVariable, Integer> varToFieldIndex = new LinkedHashMap<>();
+ Map<LogicalVariable, IAType> varTypes = new LinkedHashMap<>();
+
+ int numSecondaryKeys = idx.isQuantized() ? VTreeDataTupleAccessor.Q_NUM_SECONDARY_FIELDS
+ : VTreeDataTupleAccessor.NQ_NUM_SECONDARY_FIELDS;
+ int fieldIndex = numSecondaryKeys + idx.numPrimaryKeys();
+ for (List<String> fieldPath : includeFieldNames) {
+ LogicalVariable var = varSupplier.get();
+ // An open field not in the type declaration has no declared type; ANY still lets the type
+ // environment resolve the variable.
+ IAType fieldType = idx.recordType().getSubFieldType(fieldPath);
+ if (fieldType == null) {
+ fieldType = BuiltinType.ANY;
+ }
+ pathToVar.put(fieldPath, var);
+ varToFieldIndex.put(var, fieldIndex++);
+ varTypes.put(var, fieldType);
+ }
+ // Every column is still declared -- the tuple the runtime writes has to match the declaration either
+ // way -- but two identical INCLUDE paths leave no unambiguous variable to bind a reference to, so
+ // nothing is bound and the predicate stays above the search. DDL rejects such an index
+ // (INDEX_ILLEGAL_REPETITIVE_FIELD), so this is a belt-and-braces check, not a reachable path.
+ if (pathToVar.size() < includeFieldNames.size()) {
+ pathToVar.clear();
+ }
+ return new IncludeColumns(pathToVar, varToFieldIndex, varTypes);
+ }
+
+ /** The INCLUDE columns declared on {@code vectorUnnest}, or {@code null} if it has none. */
+ public static IncludeColumns getIncludeColumns(UnnestMapOperator vectorUnnest) {
+ return (IncludeColumns) vectorUnnest.getAnnotations().get(VECTOR_INCLUDE_COLUMNS);
}
/**
- * Whether {@code vectorUnnest} already has its INCLUDE filter variables declared -- i.e. the index-only
- * plan bound a predicate to them during the access-method phase and left it in a SELECT for
- * {@link PushFilterIntoVectorSearchRule} to move in.
+ * Rewrite {@code condition} to read the declared INCLUDE columns instead of the record, or return
+ * {@code null} when any part of it cannot be served that way.
+ *
+ * @param selectOp the {@code SELECT} carrying the condition; the {@code ASSIGN} chain below it is inlined
+ * so that field accesses on the record are visible
*/
- public static boolean hasDeclaredFilterVariables(UnnestMapOperator vectorUnnest) {
- return vectorUnnest.getAnnotations().containsKey(VECTOR_FILTER_VAR_MAPPING);
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.GENERATED)
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_FABLE_5_1, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.REFACTORED, notes = "Bind a path through its covering prefix, as a projection does")
+ public static ILogicalExpression bindPredicate(ILogicalExpression condition, ILogicalOperator selectOp,
+ IndexContext idx, IOptimizationContext context, IncludeColumns columns) throws AlgebricksException {
+ if (condition == null) {
+ return null;
+ }
+ MutableObject<ILogicalExpression> conditionRef = new MutableObject<>(condition.cloneExpression());
+ inlineAssigns(conditionRef, selectOp, context);
+
+ // Bind every field access whose path, or whose longest prefix, is an INCLUDE column: `m.info.year`
+ // over INCLUDE (info) becomes a field access on the emitted column, exactly as it does in a
+ // projection. There is no separate "is every path covered" test; the guard below is the whole
+ // decision. A predicate reading no field at all still binds, trivially, to nothing. It has to:
+ // leaving it in a SELECT above the search would filter a candidate set the search had already capped.
+ Map<List<String>, LogicalVariable> pathToVar = columns == null ? Map.of() : columns.pathToVar();
+ ILogicalExpression rewritten = rewriteFieldAccess(conditionRef.getValue(), pathToVar, idx, Map.of());
+
+ // Completeness guard: the pushed condition must reference ONLY INCLUDE column variables. A field
+ // access no INCLUDE column covers, an access pattern resolveFieldPath does not recognize, or a
+ // non-inlinable ASSIGN output each leave a reference to the source record behind -- a variable that
+ // is not in scope at the unnest.
+ Set<LogicalVariable> rewrittenUsed = new HashSet<>();
+ rewritten.getUsedVariables(rewrittenUsed);
+ if (!new HashSet<>(pathToVar.values()).containsAll(rewrittenUsed)) {
+ return null;
+ }
+ return rewritten;
}
/**
- * A predicate that has been fully rewritten against a vector index's INCLUDE columns.
+ * Replace, in place, every field access under {@code exprRef} whose full path is a key of
+ * {@code pathToVar} with a reference to that variable.
+ * <p>
+ * The same rewrite {@link #bindPredicate} applies, without its guard: a projection legitimately reads
+ * other things too, which are left alone. An access whose own path does not match is descended into, so
+ * a query reading {@code m.info.year} still binds when the column is {@code INCLUDE (info)}.
+ * <p>
+ * Paths, not names, so this serves a nested column exactly as it serves a top-level one -- and it is the
+ * same operation whether the variable carries an INCLUDE column or a primary key.
*
- * @param condition the rewritten predicate, referencing only the variables in {@code varToFieldIndex}
- * @param varToFieldIndex each fresh filter variable's physical field index in the secondary tuple
- * @param varTypes each fresh filter variable's type, for the filter's type environment
+ * @param bindings the plan's ASSIGN bindings, through which an access split across ASSIGNs
+ * ({@code $$a := m.info; $$b := $$a.year}) resolves to its full path; see
+ * {@link #resolveRecordFieldPath(AbstractFunctionCallExpression, IndexContext, Map)}
+ */
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.GENERATED)
+ public static void bindPaths(Mutable<ILogicalExpression> exprRef, IndexContext idx,
+ Map<List<String>, LogicalVariable> pathToVar, Map<LogicalVariable, ILogicalExpression> bindings)
+ throws AlgebricksException {
+ if (pathToVar == null || pathToVar.isEmpty()) {
+ return;
+ }
+ exprRef.setValue(rewriteFieldAccess(exprRef.getValue(), pathToVar, idx, bindings));
+ }
+
+ /**
+ * Move the predicate into the unnest-map's select condition, which is where the runtime reads it from.
+ * Only {@link PushFilterIntoVectorSearchRule} calls this, during the physical rewrites: a select
+ * condition reads variables the operator produces rather than variables from its input, which is not what
+ * an expression on an operator normally means, so it is created as late as possible to keep it out of the
+ * way of the rules that walk expressions.
*/
- public record PushedIncludeFilter(ILogicalExpression condition, Map<LogicalVariable, Integer> varToFieldIndex,
- Map<LogicalVariable, IAType> varTypes) {
+ public static void apply(UnnestMapOperator vectorUnnest, ILogicalExpression condition) {
+ vectorUnnest.setSelectCondition(new MutableObject<>(condition));
}
/**
@@ -158,102 +259,20 @@ public final class VectorIncludeFilterPushdown {
}
/**
- * Whether {@code condition} can be pushed, without allocating any variables or touching the
- * optimization context. Used by the cost model and by the index-only gate.
+ * Whether {@code condition} can be served entirely from the index's INCLUDE columns. Used by the
+ * index-only gate, which has to decide before any of those columns exist.
+ * <p>
+ * A dry run of {@link #bindPredicate} against throwaway variables, rather than a second implementation
+ * of the same checks. The gate's answer is irreversible -- {@code indexOnly} is serialized into the
+ * index-search arguments before the binding runs, so a gate that admits a predicate the binding then
+ * declines leaves the plan unable to rebind it -- which is exactly why the two must not be able to
+ * drift apart.
*/
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.REFACTORED)
public static boolean isPushable(ILogicalExpression condition, ILogicalOperator selectOp, IndexContext idx,
IOptimizationContext context) throws AlgebricksException {
- return analyze(condition, selectOp, idx, context, new ThrowawayVariableSupplier()) != null;
- }
-
- /**
- * Rewrites {@code condition} to read the index's INCLUDE columns, or returns {@code null} when any part
- * of it cannot be served that way.
- *
- * @param condition the {@code SELECT} condition to push
- * @param selectOp the {@code SELECT} carrying {@code condition}; the {@code ASSIGN} chain below it is
- * inlined into the condition so that field accesses on the record are visible
- * @param varSupplier allocates the fresh filter variables — {@code context::newVar} when the result
- * will be spliced into the plan, a throwaway supplier when only the verdict matters
- */
- public static PushedIncludeFilter analyze(ILogicalExpression condition, ILogicalOperator selectOp, IndexContext idx,
- IOptimizationContext context, Supplier<LogicalVariable> varSupplier) throws AlgebricksException {
- if (condition == null || idx.includeFieldNames() == null || idx.includeFieldNames().isEmpty()) {
- return null;
- }
-
- MutableObject<ILogicalExpression> conditionRef = new MutableObject<>(condition.cloneExpression());
- inlineAssigns(conditionRef, selectOp, context);
-
- // Collect the FULL field paths the filter reads, resolved against the record variable(s) this vector
- // search feeds. Matching on full paths (not leaf names) is what keeps `WHERE m.b.year > 2000` from
- // being pushed against an `INCLUDE (a.year)` column, and restricting the base to this subtree's
- // record variable(s) is what keeps a second record (from a join or a LET) from being redirected into
- // the index's INCLUDE field.
- Set<List<String>> filterFieldPaths = new HashSet<>();
- extractFieldPaths(conditionRef.getValue(), idx, filterFieldPaths);
- if (filterFieldPaths.isEmpty()) {
- return null;
- }
-
- // Bail out if the filter references any field that is not in the index's INCLUDE list.
- Map<List<String>, Integer> includeFieldIndex = buildIncludeFieldIndex(idx.includeFieldNames());
- // Two identical INCLUDE paths would collapse the map and bind the filter to whichever was inserted
- // last; bail rather than risk pushing a predicate that resolves to the wrong physical field.
- if (includeFieldIndex.size() < idx.includeFieldNames().size()) {
- return null;
- }
- for (List<String> fieldPath : filterFieldPaths) {
- if (!includeFieldIndex.containsKey(fieldPath)) {
- return null;
- }
- }
-
- // Physical tuple format depends on quantization:
- // Non-quantized: [distance, centroidId, pk_0..pk_{N-1}, include_fields...]
- // Quantized: [distance, centroidId, qDist, qEmbed, pk_0..pk_{N-1}, include_fields...]
- // INCLUDE fields start after the secondary keys and ALL primary key columns.
- int numSecondaryKeys = idx.isQuantized() ? VTreeDataTupleAccessor.Q_NUM_SECONDARY_FIELDS
- : VTreeDataTupleAccessor.NQ_NUM_SECONDARY_FIELDS;
- int includeFieldPhysicalIndex = numSecondaryKeys + idx.numPrimaryKeys();
-
- Map<List<String>, LogicalVariable> fieldToNewVar = new HashMap<>();
- // Insertion-ordered (INCLUDE order): the caller appends these to the unnest-map's variable list,
- // so a hash order would make the emitted plan differ run to run for a multi-field predicate.
- Map<LogicalVariable, Integer> filterVarToFieldIndex = new LinkedHashMap<>();
- Map<LogicalVariable, IAType> filterVarToType = new LinkedHashMap<>();
-
- for (List<String> fieldPath : idx.includeFieldNames()) {
- if (filterFieldPaths.contains(fieldPath)) {
- LogicalVariable newVar = varSupplier.get();
- fieldToNewVar.put(fieldPath, newVar);
- filterVarToFieldIndex.put(newVar, includeFieldPhysicalIndex);
-
- // For open-schema fields not in the type declaration, getSubFieldType returns null;
- // default to ANY so the type environment can still resolve the variable type.
- IAType fieldType = idx.recordType().getSubFieldType(fieldPath);
- if (fieldType == null) {
- fieldType = BuiltinType.ANY;
- }
- filterVarToType.put(newVar, fieldType);
- }
- includeFieldPhysicalIndex++;
- }
-
- ILogicalExpression rewritten = rewriteFieldAccess(conditionRef.getValue(), fieldToNewVar, idx);
-
- // Completeness guard: the pushed condition must reference ONLY the freshly-created INCLUDE
- // variables. The earlier checks act on field paths that resolveFieldPath recognizes; an access
- // pattern it does not recognize, or a non-inlinable ASSIGN output, can leave a reference to the
- // source record in the condition. Embedding such a condition would reference a variable that is not
- // in scope at the unnest -- an invalid plan. Decline instead of emitting one.
- Set<LogicalVariable> rewrittenUsed = new HashSet<>();
- rewritten.getUsedVariables(rewrittenUsed);
- if (!new HashSet<>(fieldToNewVar.values()).containsAll(rewrittenUsed)) {
- return null;
- }
-
- return new PushedIncludeFilter(rewritten, filterVarToFieldIndex, filterVarToType);
+ return bindPredicate(condition, selectOp, idx, context,
+ buildIncludeColumns(idx, new ThrowawayVariableSupplier())) != null;
}
/**
@@ -313,43 +332,59 @@ public final class VectorIncludeFilterPushdown {
}
/**
- * Builds a map from an INCLUDE field's full path to its index in the INCLUDE list.
+ * The full field path {@code funcExpr} reads off one of {@code idx}'s record variables, or {@code null}
+ * if it is not a field access on one of them.
* <p>
- * Keyed by the whole path, not the leaf name: {@code INCLUDE (a.year)} must not match a filter on
- * {@code b.year}. {@link List} equality gives exactly path equality.
+ * Full paths throughout: a nested column is addressed as {@code [info, year]} and never matches a
+ * top-level {@code year}, and both access forms resolve, so callers do not have to know whether
+ * {@code ByNameToByIndexFieldAccessRule} has run.
*/
- private static Map<List<String>, Integer> buildIncludeFieldIndex(List<List<String>> includeFieldNames) {
- Map<List<String>, Integer> result = new HashMap<>();
- for (int i = 0; i < includeFieldNames.size(); i++) {
- result.put(includeFieldNames.get(i), i);
- }
- return result;
+ public static List<String> resolveRecordFieldPath(AbstractFunctionCallExpression funcExpr, IndexContext idx)
+ throws AlgebricksException {
+ return resolveFieldPath(funcExpr, idx, Map.of());
}
/**
- * Collects the full field paths the filter reads from this vector search's record variable(s).
+ * As {@link #resolveRecordFieldPath(AbstractFunctionCallExpression, IndexContext)}, also following an
+ * access whose base is a variable that {@code bindings} defines as a field access itself.
* <p>
- * Only accesses rooted at one of {@code idx.recordVars()} are collected. An access on any other record
- * is deliberately ignored here, so it survives the rewrite and trips the completeness guard in
- * {@link #analyze} — which then declines instead of silently evaluating someone else's predicate
- * against an INCLUDE column.
+ * The compiler routinely splits a nested access across ASSIGNs -- {@code $$a := m.info} then
+ * {@code $$b := $$a.year} -- so read in place, the outer access has a variable for its base and no path
+ * at all, while the inner one reads a column prefix that is not itself a column. Followed through the
+ * binding, the two are one access to {@code [info, year]}. The predicate side sidesteps this by inlining
+ * the chain first; the projection side cannot rewrite the plan to look, so it resolves through the
+ * bindings instead, and the index-only rewrite must do the same or it would leave the outer access
+ * unbound and let the neutralized chain collapse the value to MISSING.
+ *
+ * @param bindings ASSIGN bindings of the plan, variable to defining expression; may be empty
*/
- private static void extractFieldPaths(ILogicalExpression expr, IndexContext idx, Set<List<String>> fieldPaths)
- throws AlgebricksException {
- if (expr.getExpressionTag() != LogicalExpressionTag.FUNCTION_CALL) {
- return;
- }
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_FABLE_5_1, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.GENERATED, notes = "Resolve a nested access split across ASSIGNs to its full path")
+ public static List<String> resolveRecordFieldPath(AbstractFunctionCallExpression funcExpr, IndexContext idx,
+ Map<LogicalVariable, ILogicalExpression> bindings) throws AlgebricksException {
+ return resolveFieldPath(funcExpr, idx, bindings);
+ }
- AbstractFunctionCallExpression funcExpr = (AbstractFunctionCallExpression) expr;
- List<String> path = resolveFieldPath(funcExpr, idx);
- if (path != null) {
- fieldPaths.add(path);
- // Do not recurse: the nested accesses under this one are the prefix of the path just added.
+ /** The {@code variable -> expression} bindings of every ASSIGN under {@code root}, {@code root} included. */
+ public static Map<LogicalVariable, ILogicalExpression> collectAssignBindings(ILogicalOperator root) {
+ Map<LogicalVariable, ILogicalExpression> bindings = new HashMap<>();
+ collectAssignBindings(root, bindings);
+ return bindings;
+ }
+
+ private static void collectAssignBindings(ILogicalOperator op, Map<LogicalVariable, ILogicalExpression> bindings) {
+ if (op == null) {
return;
}
-
- for (Mutable<ILogicalExpression> arg : funcExpr.getArguments()) {
- extractFieldPaths(arg.getValue(), idx, fieldPaths);
+ if (op.getOperatorTag() == LogicalOperatorTag.ASSIGN) {
+ AssignOperator assign = (AssignOperator) op;
+ List<LogicalVariable> vars = assign.getVariables();
+ List<Mutable<ILogicalExpression>> exprs = assign.getExpressions();
+ for (int i = 0; i < vars.size(); i++) {
+ bindings.put(vars.get(i), exprs.get(i).getValue());
+ }
+ }
+ for (Mutable<ILogicalOperator> input : op.getInputs()) {
+ collectAssignBindings(input.getValue(), bindings);
}
}
@@ -364,8 +399,8 @@ public final class VectorIncludeFilterPushdown {
* this runs in two phases: before {@code ByNameToByIndexFieldAccessRule} (the index-only gate) accesses
* are by name, after it (the physical pushdown) a declared type's accesses are by index.
*/
- private static List<String> resolveFieldPath(AbstractFunctionCallExpression funcExpr, IndexContext idx)
- throws AlgebricksException {
+ private static List<String> resolveFieldPath(AbstractFunctionCallExpression funcExpr, IndexContext idx,
+ Map<LogicalVariable, ILogicalExpression> bindings) throws AlgebricksException {
FunctionIdentifier fid = funcExpr.getFunctionIdentifier();
boolean byName = fid.equals(BuiltinFunctions.FIELD_ACCESS_BY_NAME);
boolean byIndex = fid.equals(BuiltinFunctions.FIELD_ACCESS_BY_INDEX);
@@ -377,12 +412,22 @@ public final class VectorIncludeFilterPushdown {
List<String> prefix;
if (base.getExpressionTag() == LogicalExpressionTag.VARIABLE) {
LogicalVariable baseVar = ((VariableReferenceExpression) base).getVariableReference();
- if (!idx.recordVars().contains(baseVar)) {
- return null;
+ if (idx.recordVars().contains(baseVar)) {
+ prefix = List.of();
+ } else {
+ // A variable an ASSIGN defines as a field access is that access: the chain the compiler split
+ // is followed back to the record. Anything else is a base this search knows nothing about.
+ ILogicalExpression bound = bindings.get(baseVar);
+ if (bound == null || bound.getExpressionTag() != LogicalExpressionTag.FUNCTION_CALL) {
+ return null;
+ }
+ prefix = resolveFieldPath((AbstractFunctionCallExpression) bound, idx, bindings);
+ if (prefix == null) {
+ return null;
+ }
}
- prefix = List.of();
} else if (base.getExpressionTag() == LogicalExpressionTag.FUNCTION_CALL) {
- prefix = resolveFieldPath((AbstractFunctionCallExpression) base, idx);
+ prefix = resolveFieldPath((AbstractFunctionCallExpression) base, idx, bindings);
if (prefix == null) {
return null;
}
@@ -431,11 +476,14 @@ public final class VectorIncludeFilterPushdown {
}
/**
- * Rewrites field-access expressions to use the new INCLUDE field variables.
- * Example: {@code gt($row.getField(2), 2000)} -> {@code gt($year, 2000)}
+ * Rewrites field-access expressions to use the INCLUDE column variables. An access whose own path is not
+ * a column is descended into, so the longest covered prefix binds.
+ * Example: {@code gt($row.getField(2), 2000)} -> {@code gt($year, 2000)};
+ * {@code gt($row.info.year, 2000)} over {@code INCLUDE (info)} -> {@code gt($info.year, 2000)}
*/
private static ILogicalExpression rewriteFieldAccess(ILogicalExpression expr,
- Map<List<String>, LogicalVariable> fieldToVar, IndexContext idx) throws AlgebricksException {
+ Map<List<String>, LogicalVariable> fieldToVar, IndexContext idx,
+ Map<LogicalVariable, ILogicalExpression> bindings) throws AlgebricksException {
if (expr.getExpressionTag() != LogicalExpressionTag.FUNCTION_CALL) {
return expr;
}
@@ -444,7 +492,7 @@ public final class VectorIncludeFilterPushdown {
// Replace a field access with a variable reference only when its FULL path — resolved against this
// search's record variable — is one of the INCLUDE fields we created a variable for.
- List<String> path = resolveFieldPath(funcExpr, idx);
+ List<String> path = resolveFieldPath(funcExpr, idx, bindings);
if (path != null && fieldToVar.containsKey(path)) {
LogicalVariable newVar = fieldToVar.get(path);
VariableReferenceExpression varRef = new VariableReferenceExpression(newVar);
@@ -455,7 +503,7 @@ public final class VectorIncludeFilterPushdown {
List<Mutable<ILogicalExpression>> newArgs = new ArrayList<>();
boolean changed = false;
for (Mutable<ILogicalExpression> argRef : funcExpr.getArguments()) {
- ILogicalExpression newArg = rewriteFieldAccess(argRef.getValue(), fieldToVar, idx);
+ ILogicalExpression newArg = rewriteFieldAccess(argRef.getValue(), fieldToVar, idx, bindings);
newArgs.add(new MutableObject<>(newArg));
if (newArg != argRef.getValue()) {
changed = true;
diff --git a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IntroduceTopKAccessMethodRule.java b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IntroduceTopKAccessMethodRule.java
index 78895e7748..ad00944e34 100644
--- a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IntroduceTopKAccessMethodRule.java
+++ b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IntroduceTopKAccessMethodRule.java
@@ -37,16 +37,12 @@ import org.apache.asterix.metadata.declared.MetadataProvider;
import org.apache.asterix.metadata.entities.Index;
import org.apache.asterix.metadata.utils.DatasetUtil;
import org.apache.asterix.metadata.utils.KeyFieldTypeUtil;
-import org.apache.asterix.om.base.AString;
-import org.apache.asterix.om.base.IAObject;
-import org.apache.asterix.om.constants.AsterixConstantValue;
-import org.apache.asterix.om.functions.BuiltinFunctions;
+import org.apache.asterix.om.types.ARecordType;
import org.apache.asterix.om.types.IAType;
import org.apache.asterix.om.utils.ConstantExpressionUtil;
import org.apache.asterix.optimizer.cost.VectorIndexGeometry;
import org.apache.asterix.optimizer.rules.VectorIncludeFilterPushdown;
import org.apache.commons.lang3.mutable.Mutable;
-import org.apache.commons.lang3.mutable.MutableObject;
import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException;
import org.apache.hyracks.algebricks.common.utils.Pair;
import org.apache.hyracks.algebricks.core.algebra.base.ILogicalExpression;
@@ -56,8 +52,6 @@ import org.apache.hyracks.algebricks.core.algebra.base.LogicalExpressionTag;
import org.apache.hyracks.algebricks.core.algebra.base.LogicalOperatorTag;
import org.apache.hyracks.algebricks.core.algebra.base.LogicalVariable;
import org.apache.hyracks.algebricks.core.algebra.expressions.AbstractFunctionCallExpression;
-import org.apache.hyracks.algebricks.core.algebra.expressions.ConstantExpression;
-import org.apache.hyracks.algebricks.core.algebra.expressions.IAlgebricksConstantValue;
import org.apache.hyracks.algebricks.core.algebra.expressions.IVariableTypeEnvironment;
import org.apache.hyracks.algebricks.core.algebra.expressions.VariableReferenceExpression;
import org.apache.hyracks.algebricks.core.algebra.functions.FunctionIdentifier;
@@ -70,6 +64,7 @@ import org.apache.hyracks.algebricks.core.algebra.operators.logical.OrderOperato
import org.apache.hyracks.algebricks.core.algebra.operators.logical.SelectOperator;
import org.apache.hyracks.algebricks.core.algebra.operators.logical.visitors.VariableUtilities;
import org.apache.hyracks.algebricks.core.algebra.util.OperatorPropertiesUtil;
+import org.apache.hyracks.util.annotations.AiProvenance;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -133,12 +128,11 @@ public class IntroduceTopKAccessMethodRule extends AbstractIntroduceAccessMethod
protected double queryMinProbeFraction = 0;
protected int queryKMultiplier = 0;
- // SELECT operator info for filter pushdown
+ // The WHERE between the ORDER and the scan, and how many SELECTs carry it. The pushdown installs one
+ // condition on the search, so a subtree holding more than one SELECT is refused an index outright rather
+ // than have one of them left above the search; see chooseVectorIndex.
protected SelectOperator selectOp = null;
- protected Set<List<String>> filterFieldNames = null; // Field names referenced in WHERE clause
- // Set when the WHERE reads a field no INCLUDE list can cover. An access on the meta record resolves to
- // a bare leaf name, which would otherwise be matched against the data record's INCLUDE fields.
- protected boolean filterReadsUncoverableField = false;
+ protected int numSelectOps = 0;
/**
* Master switch for the index-only ANN plan optimization. Enabled by default: when the projection above
@@ -388,7 +382,7 @@ public class IntroduceTopKAccessMethodRule extends AbstractIntroduceAccessMethod
}
// 4. Find SELECT operator (if any) and extract filter fields.
- // Populates selectOp and filterFieldNames if a SELECT exists.
+ // Populates selectOp if a SELECT exists.
// Must be called AFTER setDatasetAndTypeMetadata so recordType is available.
findSelectOperatorInSubTree();
@@ -402,18 +396,21 @@ public class IntroduceTopKAccessMethodRule extends AbstractIntroduceAccessMethod
// 7. Choose best vector index (considering INCLUDE fields if filter exists), minus the ones the
// cost-based optimizer already ruled out.
- chooseVectorIndex(analyzedAMs, chosenIndexes);
+ chooseVectorIndex(analyzedAMs, chosenIndexes, context);
return true;
}
/**
- * Finds SELECT operator between ORDER and DATASOURCE_SCAN.
- * Also extracts filter field names from the SELECT condition.
+ * Finds the SELECT operators between ORDER and DATASOURCE_SCAN, setting {@link #selectOp} to the topmost
+ * and {@link #numSelectOps} to how many there are.
*
- * @return The SELECT operator if found, null otherwise
+ * @return The topmost SELECT operator if any, null otherwise
*/
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_FABLE_5_1, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.ASSISTED, notes = "Count every SELECT rather than stop at the first")
protected SelectOperator findSelectOperatorInSubTree() {
+ selectOp = null;
+ numSelectOps = 0;
if (orderOp.getInputs().isEmpty()) {
return null;
}
@@ -423,9 +420,10 @@ public class IntroduceTopKAccessMethodRule extends AbstractIntroduceAccessMethod
while (currentOp != null) {
if (currentOp.getOperatorTag() == LogicalOperatorTag.SELECT) {
- selectOp = (SelectOperator) currentOp;
- filterFieldNames = extractFilterFieldsFromCondition(selectOp.getCondition().getValue());
- return selectOp;
+ numSelectOps++;
+ if (selectOp == null) {
+ selectOp = (SelectOperator) currentOp;
+ }
}
if (currentOp.getOperatorTag() == LogicalOperatorTag.DATASOURCESCAN) {
@@ -437,224 +435,7 @@ public class IntroduceTopKAccessMethodRule extends AbstractIntroduceAccessMethod
currentOp = (AbstractLogicalOperator) currentOp.getInputs().get(0).getValue();
}
- return null;
- }
-
- /**
- * Extracts field names referenced in a filter condition expression.
- * Traverses function call expressions to find field access operations.
- *
- * @param condition The filter condition expression
- * @return Set of field names (as List<String> for nested field paths)
- */
- protected Set<List<String>> extractFilterFieldsFromCondition(ILogicalExpression condition) {
- Set<List<String>> fields = new HashSet<>();
- extractFieldsFromExpressionRecursive(condition, fields);
- return fields;
- }
-
- /**
- * Recursively extracts field names from an expression.
- * Handles function calls (AND, OR, comparison operators, field-access).
- */
- private void extractFieldsFromExpressionRecursive(ILogicalExpression expr, Set<List<String>> fields) {
- if (expr.getExpressionTag() == LogicalExpressionTag.FUNCTION_CALL) {
- AbstractFunctionCallExpression funcExpr = (AbstractFunctionCallExpression) expr;
-
- // Check if this is a field access function (e.g., field-access-by-name)
- if (funcExpr.getFunctionIdentifier().equals(BuiltinFunctions.FIELD_ACCESS_BY_NAME)
- || funcExpr.getFunctionIdentifier().equals(BuiltinFunctions.FIELD_ACCESS_BY_INDEX)) {
- if (isMetaRecordAccess(funcExpr)) {
- // Record that the filter cannot be covered rather than its name: the name belongs to the
- // meta record, and adding it would be matched against the data record's INCLUDE list --
- // `meta(m).expiration` would read the record's `expiration` column.
- filterReadsUncoverableField = true;
- return;
- }
- // Extract field name from field access function
- List<String> fieldPath = extractFieldPathFromFieldAccess(funcExpr);
- if (fieldPath != null && !fieldPath.isEmpty()) {
- fields.add(fieldPath);
- }
- } else {
- // Recursively process arguments for other functions (AND, OR, GT, LT, etc.)
- for (Mutable<ILogicalExpression> arg : funcExpr.getArguments()) {
- extractFieldsFromExpressionRecursive(arg.getValue(), fields);
- }
- }
- } else if (expr.getExpressionTag() == LogicalExpressionTag.VARIABLE) {
- // Variable reference - trace back through assigns to find field access.
- // First check subTree assigns; if not found, scan from ORDER down to DATASOURCE_SCAN.
- VariableReferenceExpression varRef = (VariableReferenceExpression) expr;
- LogicalVariable var = varRef.getVariableReference();
- boolean found = findFieldFromAssigns(var, subTree.getAssignsAndUnnests(), fields);
- if (!found && orderOp != null) {
- searchAssignsInPlan(var, orderOp, fields);
- }
- }
- }
-
- /**
- * Searches for variable definition in a list of assigns and extracts field info.
- */
- private boolean findFieldFromAssigns(LogicalVariable var, List<AbstractLogicalOperator> assigns,
- Set<List<String>> fields) {
- for (AbstractLogicalOperator op : assigns) {
- if (op.getOperatorTag() == LogicalOperatorTag.ASSIGN) {
- AssignOperator assignOp = (AssignOperator) op;
- List<LogicalVariable> assignVars = assignOp.getVariables();
- List<Mutable<ILogicalExpression>> assignExprs = assignOp.getExpressions();
-
- for (int i = 0; i < assignVars.size(); i++) {
- if (assignVars.get(i).equals(var)) {
- extractFieldsFromExpressionRecursive(assignExprs.get(i).getValue(), fields);
- return true;
- }
- }
- }
- }
- return false;
- }
-
- /**
- * Searches all operators in the plan from the given operator downward for variable definition.
- */
- private boolean searchAssignsInPlan(LogicalVariable var, ILogicalOperator startOp, Set<List<String>> fields) {
- ILogicalOperator currentOp = startOp;
-
- while (currentOp != null) {
- if (currentOp.getOperatorTag() == LogicalOperatorTag.ASSIGN) {
- AssignOperator assignOp = (AssignOperator) currentOp;
- List<LogicalVariable> assignVars = assignOp.getVariables();
- List<Mutable<ILogicalExpression>> assignExprs = assignOp.getExpressions();
-
- for (int i = 0; i < assignVars.size(); i++) {
- if (assignVars.get(i).equals(var)) {
- // Found the assignment - recursively extract fields
- extractFieldsFromExpressionRecursive(assignExprs.get(i).getValue(), fields);
- return true;
- }
- }
- }
-
- // Move to next operator
- if (currentOp.getInputs().isEmpty()) {
- break;
- }
- currentOp = currentOp.getInputs().get(0).getValue();
- }
- return false;
- }
-
- /**
- * Whether {@code funcExpr} is a field access rooted at this subtree's meta record rather than its data
- * record. The two are told apart by the subtree's own key-source mapping, the same way
- * {@link #isProjectionPkOnly} separates a meta-sourced primary key from a record field of that name.
- */
- private boolean isMetaRecordAccess(AbstractFunctionCallExpression funcExpr) {
- ILogicalExpression base = funcExpr.getArguments().isEmpty() ? null : funcExpr.getArguments().get(0).getValue();
- while (base != null && base.getExpressionTag() == LogicalExpressionTag.FUNCTION_CALL) {
- List<Mutable<ILogicalExpression>> baseArgs = ((AbstractFunctionCallExpression) base).getArguments();
- base = baseArgs.isEmpty() ? null : baseArgs.get(0).getValue();
- }
- if (base == null || base.getExpressionTag() != LogicalExpressionTag.VARIABLE || subTree == null) {
- return false;
- }
- LogicalVariable baseVar = ((VariableReferenceExpression) base).getVariableReference();
- OptimizableOperatorSubTree.RecordTypeSource varType = subTree.getRecordTypeFor(baseVar);
- return varType != null && varType.sourceIndicator == 1;
- }
-
- /**
- * Extracts field path from a field-access function expression.
- * For nested fields like row.nested.field, returns ["nested", "field"].
- */
- private List<String> extractFieldPathFromFieldAccess(AbstractFunctionCallExpression funcExpr) {
- List<String> fieldPath = new ArrayList<>();
- extractFieldPathRecursive(funcExpr, fieldPath);
- return fieldPath;
- }
-
- /**
- * Recursively builds field path from nested field access expressions.
- */
- private void extractFieldPathRecursive(ILogicalExpression expr, List<String> fieldPath) {
- if (expr.getExpressionTag() != LogicalExpressionTag.FUNCTION_CALL) {
- return;
- }
-
- AbstractFunctionCallExpression funcExpr = (AbstractFunctionCallExpression) expr;
-
- if (funcExpr.getFunctionIdentifier().equals(BuiltinFunctions.FIELD_ACCESS_BY_NAME)) {
- // First argument is the record, second argument is the field name
- if (funcExpr.getArguments().size() >= 2) {
- // Recursively process the record argument (for nested field access)
- extractFieldPathRecursive(funcExpr.getArguments().get(0).getValue(), fieldPath);
-
- // Extract field name from second argument
- ILogicalExpression fieldNameExpr = funcExpr.getArguments().get(1).getValue();
- String fieldName = AccessMethodUtils.getStringConstant(new MutableObject<>(fieldNameExpr));
- if (fieldName != null) {
- fieldPath.add(fieldName);
- }
- }
- } else if (funcExpr.getFunctionIdentifier().equals(BuiltinFunctions.FIELD_ACCESS_BY_INDEX)) {
- // First argument is the record, second argument is the field index
- if (funcExpr.getArguments().size() >= 2) {
- // Recursively process the record argument (for nested field access)
- extractFieldPathRecursive(funcExpr.getArguments().get(0).getValue(), fieldPath);
-
- // Extract field index from second argument and look up field name
- ILogicalExpression fieldIndexExpr = funcExpr.getArguments().get(1).getValue();
- Integer fieldIndex = AccessMethodUtils.getInt32Constant(new MutableObject<>(fieldIndexExpr));
- if (fieldIndex != null && subTree.getRecordType() != null) {
- String[] fieldNames = subTree.getRecordType().getFieldNames();
- if (fieldIndex >= 0 && fieldIndex < fieldNames.length) {
- fieldPath.add(fieldNames[fieldIndex]);
- }
- }
- }
- }
- }
-
- /**
- * Checks if a vector index has all required filter fields in its INCLUDE fields.
- *
- * @param index The vector index to check
- * @param filterFields The set of field names referenced in the filter
- * @return true if all filter fields are in the index's INCLUDE fields
- */
- protected boolean indexHasIncludeFields(Index index, Set<List<String>> filterFields) {
- if (filterFields == null || filterFields.isEmpty()) {
- // No filter fields - index can be used
- return true;
- }
-
- if (index.getIndexType() != IndexType.VTREE) {
- return false;
- }
-
- Index.VectorIndexDetails vectorDetails = (Index.VectorIndexDetails) index.getIndexDetails();
- List<List<String>> includeFieldNames = vectorDetails.getIncludeFieldNames();
-
- if (includeFieldNames == null || includeFieldNames.isEmpty()) {
- return false;
- }
-
- // Check if all filter fields are present in INCLUDE fields.
- for (List<String> filterField : filterFields) {
- boolean found = false;
- for (List<String> includeField : includeFieldNames) {
- if (includeField.equals(filterField)) {
- found = true;
- break;
- }
- }
- if (!found) {
- return false;
- }
- }
- return true;
+ return selectOp;
}
/**
@@ -797,24 +578,33 @@ public class IntroduceTopKAccessMethodRule extends AbstractIntroduceAccessMethod
* If the query specifies a constant distance metric that does not match the index metadata, compilation fails.
*/
protected void chooseVectorIndex(Map<IAccessMethod, AccessMethodAnalysisContext> analyzedAMs,
- List<Pair<IAccessMethod, Index>> result) throws AlgebricksException {
+ List<Pair<IAccessMethod, Index>> result, IOptimizationContext context) throws AlgebricksException {
AccessMethodAnalysisContext analysisCtx = analyzedAMs.get(VectorIndexAccessMethod.INSTANCE);
if (analysisCtx == null) {
return;
}
- // A filter no INCLUDE list can cover makes the vector index a liability: the predicate would have to
- // run above the search, after k has been counted, so LIMIT k could return fewer than k rows. Decline
- // the index and let the exact scan answer it, as an uncovered record field already does. Note this
- // cannot be folded into the hasFilter test below -- such a field contributes no name, so a predicate
- // reading nothing else leaves filterFieldNames empty and would read as "no filter at all".
- if (filterReadsUncoverableField) {
+ // A predicate that cannot be moved into the search must not leave a SELECT above it. The search
+ // caps candidates at k * k_multiplier, chosen without regard to the predicate, so filtering
+ // afterwards answers "the passing rows among the k nearest" instead of "the k nearest passing
+ // rows" -- and it can return nothing at all where passing rows exist. A non-functional predicate
+ // (random(), and anything else registered with isFunctional = false) cannot be evaluated per
+ // candidate in the cursor, so no vector index is offered and the query takes the exact scan.
+ if (selectOp != null && !selectOp.getCondition().getValue().isFunctional()) {
+ LOGGER.trace("chooseVectorIndex: the WHERE is non-functional and cannot be pushed into the "
+ + "search; not offering a vector index");
+ return;
+ }
+ // The gate and the binding each handle one SELECT, and the search takes one condition. With several,
+ // pushing the topmost would leave the others above the search -- the very shape being avoided -- so
+ // no index is offered. The select rules consolidate adjacent SELECTs and push each as far down as its
+ // variables allow, so this shape is not known to arise; it is refused rather than assumed away.
+ if (numSelectOps > 1) {
+ LOGGER.trace("chooseVectorIndex: {} SELECTs between the ORDER and the scan, of which the pushdown "
+ + "can install one; not offering a vector index", numSelectOps);
return;
}
-
- // Check if query has filter predicates
- boolean hasFilter = selectOp != null && filterFieldNames != null && !filterFieldNames.isEmpty();
// Iterate over candidate vector indexes
Iterator<Map.Entry<Index, List<Pair<Integer, Integer>>>> indexIt =
@@ -834,8 +624,15 @@ public class IntroduceTopKAccessMethodRule extends AbstractIntroduceAccessMethod
continue;
}
- // If query has filter, index must have all filter fields in its INCLUDE list.
- if (hasFilter && !indexHasIncludeFields(index, filterFieldNames)) {
+ // A predicate must be evaluable INSIDE the search, or this index cannot answer the query
+ // at all. The search caps candidates at k * k_multiplier, chosen without regard to the
+ // predicate, so a predicate left in a SELECT above it answers "the passing rows among the
+ // k nearest" instead of "the k nearest passing rows" -- silently wrong, and it can return
+ // nothing where passing rows exist. Asking the pushdown itself, rather than a separate
+ // name-matching test, is what makes this decision agree with the binding that follows: an
+ // index offered here is one PushFilterIntoVectorSearchRule can and will bind, leaving no
+ // SELECT above the search in either plan shape.
+ if (selectOp != null && !isFilterPushableToInclude(index, context)) {
continue;
}
@@ -1033,21 +830,20 @@ public class IntroduceTopKAccessMethodRule extends AbstractIntroduceAccessMethod
AbstractScanOperator dataSourceOp = (AbstractScanOperator) dataSourceOpRaw;
List<LogicalVariable> dsVars = dataSourceOp.getVariables();
- // PK fields: only support top-level single-segment paths (e.g. ["id"]); composite or nested PK
- // paths are out of scope for the initial activation. Split PK field names by their key source
- // (data record vs. meta record) so a PK sourced from meta() named "id" is never confused with an
- // unrelated data-record field that happens to share the same name.
+ // PK field PATHS, split by their key source (data record vs. meta record) so a PK sourced from
+ // meta() named "id" is never confused with an unrelated data-record field that happens to share the
+ // name. Full paths, so a nested primary key is matched exactly like a top-level one.
List<List<String>> pkPaths = subTree.getDataset().getPrimaryKeys();
List<Integer> keySourceIndicators = DatasetUtil.getKeySourceIndicators(subTree.getDataset());
- Set<String> recordPkFieldNames = new HashSet<>();
- Set<String> metaPkFieldNames = new HashSet<>();
+ Set<List<String>> recordPkFieldPaths = new HashSet<>();
+ Set<List<String>> metaPkFieldPaths = new HashSet<>();
for (int i = 0; i < pkPaths.size(); i++) {
List<String> p = pkPaths.get(i);
- if (p == null || p.size() != 1) {
+ if (p == null || p.isEmpty()) {
return false;
}
boolean fromMeta = keySourceIndicators != null && keySourceIndicators.get(i) == 1;
- (fromMeta ? metaPkFieldNames : recordPkFieldNames).add(p.get(0));
+ (fromMeta ? metaPkFieldPaths : recordPkFieldPaths).add(p);
}
int numPK = pkPaths.size();
if (dsVars.size() < numPK + 1) {
@@ -1060,12 +856,18 @@ public class IntroduceTopKAccessMethodRule extends AbstractIntroduceAccessMethod
Set<LogicalVariable> recordVars = new HashSet<>(dsVars.subList(numPK, dsVars.size()));
LogicalVariable dataRecordVar = dsVars.get(numPK);
LogicalVariable metaRecordVar = dsVars.size() > numPK + 1 ? dsVars.get(numPK + 1) : null;
- PkFieldContext pkCtx = new PkFieldContext(pkVars, recordVars, dataRecordVar, metaRecordVar, recordPkFieldNames,
- metaPkFieldNames);
+ // An INCLUDE column is served like a primary key: the search reads it from the secondary tuple and
+ // emits it, so the record never has to be assembled to return it.
+ Index.VectorIndexDetails details = (Index.VectorIndexDetails) vectorIndex.getIndexDetails();
+ List<List<String>> includePaths = details.getIncludeFieldNames();
+ PkFieldContext pkCtx = new PkFieldContext(pkVars, recordVars, dataRecordVar, metaRecordVar, recordPkFieldPaths,
+ metaPkFieldPaths, includePaths == null ? Set.of() : new HashSet<>(includePaths),
+ pathContext(subTree.getRecordType(), dataRecordVar == null ? Set.of() : Set.of(dataRecordVar)),
+ pathContext(subTree.getMetaRecordType(), metaRecordVar == null ? Set.of() : Set.of(metaRecordVar)));
// Collect ASSIGN bindings throughout the entire subtree.
- Map<LogicalVariable, ILogicalExpression> bindings = new HashMap<>();
- collectAssignBindings(subTree.getRoot(), bindings);
+ Map<LogicalVariable, ILogicalExpression> bindings =
+ VectorIncludeFilterPushdown.collectAssignBindings(subTree.getRoot());
// Gather variables used above the LIMIT.
Set<LogicalVariable> liveOut = new HashSet<>();
@@ -1104,7 +906,14 @@ public class IntroduceTopKAccessMethodRule extends AbstractIntroduceAccessMethod
return false;
}
}
- return true;
+ // Every variable the predicate reads is one the search emits, which is still not enough: the
+ // index-only branch installs a predicate by rebinding it onto the INCLUDE columns, and that is its
+ // only way to evaluate one -- the record is never assembled. A predicate it cannot rebind leaves it
+ // holding a SELECT it cannot install, which job generation then rejects. Two such predicates reach
+ // here with every variable "safe": one over primary keys, and one reading no dataset field at all
+ // (`WHERE random() > 0.5`), whose variable set is empty so the loop above passes vacuously. Both
+ // belong on lookup-and-rerank, where the SELECT can simply stay above the search.
+ return selectOp == null || isFilterPushableToInclude(vectorIndex, context);
}
/**
@@ -1176,20 +985,39 @@ public class IntroduceTopKAccessMethodRule extends AbstractIntroduceAccessMethod
final Set<LogicalVariable> recordVars;
final LogicalVariable dataRecordVar;
final LogicalVariable metaRecordVar;
- final Set<String> recordPkFieldNames;
- final Set<String> metaPkFieldNames;
+ final Set<List<String>> recordPkFieldPaths;
+ final Set<List<String>> metaPkFieldPaths;
+ /** The searched index's INCLUDE columns, which the search emits alongside the primary keys. */
+ final Set<List<String>> includePaths;
+ /** Resolves a field access on the data record, and on the meta record, to its full path. */
+ final VectorIncludeFilterPushdown.IndexContext dataRecordCtx;
+ final VectorIncludeFilterPushdown.IndexContext metaRecordCtx;
PkFieldContext(Set<LogicalVariable> pkVars, Set<LogicalVariable> recordVars, LogicalVariable dataRecordVar,
- LogicalVariable metaRecordVar, Set<String> recordPkFieldNames, Set<String> metaPkFieldNames) {
+ LogicalVariable metaRecordVar, Set<List<String>> recordPkFieldPaths, Set<List<String>> metaPkFieldPaths,
+ Set<List<String>> includePaths, VectorIncludeFilterPushdown.IndexContext dataRecordCtx,
+ VectorIncludeFilterPushdown.IndexContext metaRecordCtx) {
this.pkVars = pkVars;
this.recordVars = recordVars;
this.dataRecordVar = dataRecordVar;
this.metaRecordVar = metaRecordVar;
- this.recordPkFieldNames = recordPkFieldNames;
- this.metaPkFieldNames = metaPkFieldNames;
+ this.recordPkFieldPaths = recordPkFieldPaths;
+ this.metaPkFieldPaths = metaPkFieldPaths;
+ this.includePaths = includePaths;
+ this.dataRecordCtx = dataRecordCtx;
+ this.metaRecordCtx = metaRecordCtx;
}
}
+ /**
+ * A context that resolves a field access rooted at one of {@code recordVars} to its full path. Only
+ * the record type and the accepted bases matter for that, so the index-side fields are left empty.
+ */
+ private static VectorIncludeFilterPushdown.IndexContext pathContext(ARecordType recordType,
+ Set<LogicalVariable> recordVars) {
+ return new VectorIncludeFilterPushdown.IndexContext(null, recordType, false, 0, recordVars);
+ }
+
/**
* Collect the variables used in every {@code SELECT} condition in the subtree (below the LIMIT). Used by
* {@link #isProjectionPkOnly} to reject the index-only plan when a {@code WHERE} needs a non-PK record
@@ -1211,28 +1039,6 @@ public class IntroduceTopKAccessMethodRule extends AbstractIntroduceAccessMethod
}
}
- /**
- * Walk {@code op} and its inputs collecting {@code variable → expression} bindings from every
- * {@code ASSIGN} operator. Used by {@link #isProjectionPkOnly} to trace derived live-out variables
- * back to their definitions.
- */
- private void collectAssignBindings(ILogicalOperator op, Map<LogicalVariable, ILogicalExpression> bindings) {
- if (op == null) {
- return;
- }
- if (op.getOperatorTag() == LogicalOperatorTag.ASSIGN) {
- AssignOperator a = (AssignOperator) op;
- List<LogicalVariable> vars = a.getVariables();
- List<Mutable<ILogicalExpression>> exprs = a.getExpressions();
- for (int i = 0; i < vars.size(); i++) {
- bindings.put(vars.get(i), exprs.get(i).getValue());
- }
- }
- for (Mutable<ILogicalOperator> input : op.getInputs()) {
- collectAssignBindings(input.getValue(), bindings);
- }
- }
-
private boolean isVarPkSafe(LogicalVariable v, Map<LogicalVariable, ILogicalExpression> bindings,
PkFieldContext pkCtx, Set<LogicalVariable> visiting) {
if (pkCtx.pkVars.contains(v)) {
@@ -1274,19 +1080,31 @@ public class IntroduceTopKAccessMethodRule extends AbstractIntroduceAccessMethod
// (e.g. a data-record field "id" next to a meta()-sourced PK also named "id") must not be
// treated as PK-safe here — only field-access on the record the PK is actually declared on
// is a safe substitute.
- if (BuiltinFunctions.FIELD_ACCESS_BY_NAME.equals(fid) && fce.getArguments().size() == 2) {
- ILogicalExpression a0 = fce.getArguments().get(0).getValue();
- ILogicalExpression a1 = fce.getArguments().get(1).getValue();
- if (a0.getExpressionTag() == LogicalExpressionTag.VARIABLE
- && a1.getExpressionTag() == LogicalExpressionTag.CONSTANT) {
- LogicalVariable target = ((VariableReferenceExpression) a0).getVariableReference();
- if (target.equals(pkCtx.dataRecordVar) || target.equals(pkCtx.metaRecordVar)) {
- String fieldName = extractStringConstant(a1);
- Set<String> allowedNames = target.equals(pkCtx.dataRecordVar) ? pkCtx.recordPkFieldNames
- : pkCtx.metaPkFieldNames;
- return fieldName != null && allowedNames.contains(fieldName);
- }
+ // A field access on the record is safe when its FULL path is a primary key or, on the data
+ // record, an INCLUDE column of the searched index -- both are emitted by the search. Paths
+ // rather than names, resolved through the same code the filter pushdown uses, so a nested
+ // column and either access form (by name, or by index after
+ // ByNameToByIndexFieldAccessRule) are all handled, as is an access the compiler split across
+ // ASSIGNs ($$a := m.info; $$b := $$a.year), which resolves through the bindings to its full
+ // path. An access that does not itself match falls through to the argument walk below, which
+ // reaches a shorter path that may: reading `m.info.year` is servable by an `INCLUDE (info)`
+ // column.
+ try {
+ List<String> path =
+ VectorIncludeFilterPushdown.resolveRecordFieldPath(fce, pkCtx.dataRecordCtx, bindings);
+ if (path != null
+ && (pkCtx.recordPkFieldPaths.contains(path) || pkCtx.includePaths.contains(path))) {
+ return true;
}
+ List<String> metaPath =
+ VectorIncludeFilterPushdown.resolveRecordFieldPath(fce, pkCtx.metaRecordCtx, bindings);
+ if (metaPath != null && pkCtx.metaPkFieldPaths.contains(metaPath)) {
+ return true;
+ }
+ } catch (AlgebricksException resolveFailure) {
+ LOGGER.trace("isProjectionPkOnly: could not resolve a field access; not index-only",
+ resolveFailure);
+ return false;
}
// For any other function call: all argument expressions must be PK-safe.
for (Mutable<ILogicalExpression> arg : fce.getArguments()) {
@@ -1301,19 +1119,6 @@ public class IntroduceTopKAccessMethodRule extends AbstractIntroduceAccessMethod
}
}
- private static String extractStringConstant(ILogicalExpression e) {
- if (e.getExpressionTag() != LogicalExpressionTag.CONSTANT) {
- return null;
- }
- ConstantExpression ce = (ConstantExpression) e;
- IAlgebricksConstantValue v = ce.getValue();
- if (!(v instanceof AsterixConstantValue)) {
- return null;
- }
- IAObject obj = ((AsterixConstantValue) v).getObject();
- return (obj instanceof AString) ? ((AString) obj).getStringValue() : null;
- }
-
/**
* Clears the state for the next optimization attempt.
*/
@@ -1328,8 +1133,7 @@ public class IntroduceTopKAccessMethodRule extends AbstractIntroduceAccessMethod
queryMinProbeFraction = 0;
queryKMultiplier = 0;
selectOp = null;
- filterFieldNames = null;
- filterReadsUncoverableField = false;
+ numSelectOps = 0;
aboveLimitOps.clear();
subTree.reset();
}
diff --git a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/VectorIndexAccessMethod.java b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/VectorIndexAccessMethod.java
index da3b9c69c6..25b101e8a9 100644
--- a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/VectorIndexAccessMethod.java
+++ b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/VectorIndexAccessMethod.java
@@ -23,6 +23,7 @@ import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -38,10 +39,8 @@ import org.apache.asterix.metadata.entities.Index;
import org.apache.asterix.metadata.utils.DatasetUtil;
import org.apache.asterix.om.base.ADouble;
import org.apache.asterix.om.base.AInt32;
-import org.apache.asterix.om.base.AInt64;
import org.apache.asterix.om.base.AMissing;
import org.apache.asterix.om.base.AString;
-import org.apache.asterix.om.base.IAObject;
import org.apache.asterix.om.constants.AsterixConstantValue;
import org.apache.asterix.om.functions.BuiltinFunctions;
import org.apache.asterix.om.types.ARecordType;
@@ -86,7 +85,7 @@ import org.apache.hyracks.util.annotations.AiProvenance;
* <pre>
* SELECT id, title
* FROM movie
- * WHERE year > 2000 -- Handled by BTreeAccessMethod
+ * WHERE year > 2000 -- Evaluated inside the search when `year` is an INCLUDE column of the index
* ORDER BY ANN_DISTANCE(reviewEmbedding, [1.0, 2.0, ...], "Euclidean") -- Handled by VectorIndexAccessMethod
* LIMIT 10
* </pre>
@@ -213,7 +212,9 @@ public class VectorIndexAccessMethod implements IAccessMethod {
* - ORDER BY: Computes exact distances on full records
* - LIMIT: Extracts final top-k results
*
- * TODO: Add index-only plan optimization (skip primary lookup when only PK is needed)
+ * With {@code indexOnly} set there is no primary lookup: the search emits the primary keys, the distance
+ * and the INCLUDE columns, the ORDER BY sorts on that distance, and the plan above is rebound to those
+ * outputs.
*
* @param limitRef Reference to LIMIT operator
* @param orderRef Reference to ORDER operator
@@ -224,6 +225,7 @@ public class VectorIndexAccessMethod implements IAccessMethod {
* @param context Optimization context
* @return The transformed plan with vector index search + primary lookup, or null if transformation fails
*/
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_FABLE_5_1, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.ASSISTED, notes = "Splice the index-only search into the plan before retyping the operators above it")
public ILogicalOperator createIndexSearchPlan(Mutable<ILogicalOperator> limitRef,
Mutable<ILogicalOperator> orderRef, AbstractFunctionCallExpression annDistanceExpr,
OptimizableOperatorSubTree subTree, Index chosenIndex, AccessMethodAnalysisContext analysisCtx,
@@ -319,9 +321,24 @@ public class VectorIndexAccessMethod implements IAccessMethod {
ILogicalOperator secondaryIndexUnnestOp = AccessMethodUtils.createSecondaryIndexUnnestMap(dataset, recordType,
metaRecordType, chosenIndex, assignSearchKeys, jobGenParams, context, false, false, null);
- // INCLUDE-field filter pushdown is intentionally deferred to PushFilterIntoVectorSearchRule (a
- // physical rewrite): the field-access predicate references the record variable, which is not in this
- // unnest's input type environment. Just register the variables the vector index search produces.
+ // Output order is [pk..., distance?, INCLUDE columns...]. The distance goes on first because the
+ // INCLUDE columns must be the trailing outputs: job generation lines the emitted tuple up with the
+ // output record descriptor by taking them from the end of the variable list.
+ LogicalVariable distVar = null;
+ VectorIncludeFilterPushdown.IncludeColumns includeColumns = null;
+ if (secondaryIndexUnnestOp instanceof UnnestMapOperator) {
+ UnnestMapOperator searchUnnest = (UnnestMapOperator) secondaryIndexUnnestOp;
+ if (isIndexOnlyPlan) {
+ distVar = context.newVar();
+ searchUnnest.getVariables().add(distVar);
+ searchUnnest.getVariableTypes().add(BuiltinType.ADOUBLE);
+ }
+ // Every INCLUDE column of the index is declared, for both plan shapes. One mapping then serves
+ // whatever reads them -- a predicate pushed into the search, a projection returning them, or
+ // neither -- and the runtime emits exactly what is declared.
+ includeColumns = VectorIncludeFilterPushdown.declareIncludeColumns(searchUnnest,
+ includeContext(chosenIndex, dataset, recordType, dataSourceOp), context::newVar);
+ }
context.computeAndSetTypeEnvironmentForOperator(secondaryIndexUnnestOp);
if (isIndexOnlyPlan) {
@@ -352,9 +369,6 @@ public class VectorIndexAccessMethod implements IAccessMethod {
+ "search but got " + secondaryIndexUnnestOp.getOperatorTag());
} else {
UnnestMapOperator unnestMap = (UnnestMapOperator) secondaryIndexUnnestOp;
- LogicalVariable distVar = context.newVar();
- unnestMap.getVariables().add(distVar);
- unnestMap.getVariableTypes().add(BuiltinType.ADOUBLE);
// Rewrite ORDER BY: replace the ann_distance call with VarRef($$dist). isIndexOnlyPlan was
// only left true if orderOp has exactly one expression (the ann_distance call); keep the same
@@ -367,7 +381,8 @@ public class VectorIndexAccessMethod implements IAccessMethod {
// Rebind the WHERE onto the index's INCLUDE columns. There is no primary lookup here to
// evaluate it against, and the record variable is about to be neutralized, so a predicate
// left reading the record would collapse to select(missing) and drop every row.
- bindIncludeFilterToSearch(unnestMap, selectOp, chosenIndex, dataset, recordType, dataSourceOp, context);
+ bindIncludeFilterToSearch(selectOp, chosenIndex, dataset, recordType, dataSourceOp, includeColumns,
+ context);
// Build the variable substitution map: old PK vars → new PK vars. The secondary
// UnnestMap allocates fresh context.newVar() PK variables, so anything that
@@ -375,7 +390,7 @@ public class VectorIndexAccessMethod implements IAccessMethod {
List<LogicalVariable> oldVars = dataSourceOp.getVariables();
int numPK = dataset.getPrimaryKeys().size();
List<LogicalVariable> newPkVars = unnestMap.getVariables().subList(0, numPK);
- Map<LogicalVariable, LogicalVariable> pkSubstitution = new java.util.HashMap<>();
+ Map<LogicalVariable, LogicalVariable> pkSubstitution = new HashMap<>();
for (int i = 0; i < numPK; i++) {
pkSubstitution.put(oldVars.get(i), newPkVars.get(i));
}
@@ -387,17 +402,19 @@ public class VectorIndexAccessMethod implements IAccessMethod {
LogicalVariable oldRecVar = numPK < oldVars.size() ? oldVars.get(numPK) : null;
LogicalVariable oldMetaVar = numPK + 1 < oldVars.size() ? oldVars.get(numPK + 1) : null;
- // Map PK field name -> new PK variable, split by the record the PK field is actually
- // sourced from (0 = data record, 1 = meta record).
+ // Map PK field PATH -> new PK variable, split by the record the PK field is actually
+ // sourced from (0 = data record, 1 = meta record) so a PK sourced from meta() named "id" is
+ // never confused with an unrelated data-record field of that name. Paths, not names: a
+ // nested primary key binds exactly as a top-level one does.
List<Integer> keySourceIndicators = DatasetUtil.getKeySourceIndicators(dataset);
- Map<String, LogicalVariable> recordPkFieldToNewVar = new java.util.HashMap<>();
- Map<String, LogicalVariable> metaPkFieldToNewVar = new java.util.HashMap<>();
+ Map<List<String>, LogicalVariable> recordPkPathToNewVar = new LinkedHashMap<>();
+ Map<List<String>, LogicalVariable> metaPkPathToNewVar = new LinkedHashMap<>();
List<List<String>> pkPaths = dataset.getPrimaryKeys();
for (int i = 0; i < numPK; i++) {
List<String> p = pkPaths.get(i);
- if (p != null && p.size() == 1) {
+ if (p != null && !p.isEmpty()) {
boolean fromMeta = keySourceIndicators != null && keySourceIndicators.get(i) == 1;
- (fromMeta ? metaPkFieldToNewVar : recordPkFieldToNewVar).put(p.get(0), newPkVars.get(i));
+ (fromMeta ? metaPkPathToNewVar : recordPkPathToNewVar).put(p, newPkVars.get(i));
}
}
@@ -416,11 +433,26 @@ public class VectorIndexAccessMethod implements IAccessMethod {
// 2. Rewrite field-access on the old record/meta vars to direct PK VarRefs (e.g. the
// SELECT VALUE m.idx projection above LIMIT), each against only its own PK fields.
+ // Both the primary keys and the INCLUDE columns are outputs of the search now, so pointing
+ // the plan above at them is one operation over full paths, applied to the record the path is
+ // actually declared on.
+ // Through the plan's ASSIGN bindings, so an access the compiler split across ASSIGNs binds
+ // as the one path it is -- the same resolution the gate accepted it by. Collected before any
+ // rewrite, so each variable still maps to the access it was defined as.
+ Map<LogicalVariable, ILogicalExpression> bindings =
+ VectorIncludeFilterPushdown.collectAssignBindings(rewriteRoot);
if (oldRecVar != null) {
- rewriteRecordFieldAccessToPk(rewriteRoot, oldRecVar, recordPkFieldToNewVar);
+ VectorIncludeFilterPushdown.IndexContext recordCtx =
+ includeContext(chosenIndex, dataset, recordType, Set.of(oldRecVar));
+ if (includeColumns != null) {
+ bindPathsInDescendants(rewriteRoot, recordCtx, includeColumns.pathToVar(), bindings);
+ }
+ bindPathsInDescendants(rewriteRoot, recordCtx, recordPkPathToNewVar, bindings);
}
if (oldMetaVar != null) {
- rewriteRecordFieldAccessToPk(rewriteRoot, oldMetaVar, metaPkFieldToNewVar);
+ bindPathsInDescendants(rewriteRoot,
+ includeContext(chosenIndex, dataset, metaRecordType, Set.of(oldMetaVar)),
+ metaPkPathToNewVar, bindings);
}
// 3. Any remaining ASSIGN expression that still references oldRecVar/oldMetaVar
@@ -440,17 +472,6 @@ public class VectorIndexAccessMethod implements IAccessMethod {
neutralizeDanglingExpressions(rewriteRoot, deadRecordVars);
}
- // Recompute type env bottom-up over every operator whose schema/expressions changed:
- // the new secondary UnnestMap, the LIMIT chain, then each above-LIMIT ancestor from the
- // one closest to LIMIT up to the root (aboveLimitOps is ordered root-first).
- context.computeAndSetTypeEnvironmentForOperator(secondaryIndexUnnestOp);
- org.apache.hyracks.algebricks.core.algebra.util.OperatorPropertiesUtil.typeOpRec(limitRef, context);
- if (aboveLimitOps != null) {
- for (int ai = aboveLimitOps.size() - 1; ai >= 0; ai--) {
- context.computeAndSetTypeEnvironmentForOperator(aboveLimitOps.get(ai));
- }
- }
-
// Cross-pollination dedup (index-only branch): when cross_pollination_m > 1 the secondary
// cursor emits up to M (pk..., dist) copies per record. The primary-lookup path dedups via a
// DistinctOperator above its primary UNNEST-MAP (see createRestOfIndexSearchPlan below), but the
@@ -459,6 +480,7 @@ public class VectorIndexAccessMethod implements IAccessMethod {
// so LIMIT applies to DISTINCT PKs. DistinctOperator propagates all input vars, so $$dist
// survives for the ORDER BY $$dist above it. For M == 1 we return the bare UNNEST-MAP so the
// plan stays byte-identical to the legacy path.
+ ILogicalOperator indexOnlyPlan = secondaryIndexUnnestOp;
if (extractCrossPollinationM(chosenIndex) > 1 && numPK > 0) {
List<Mutable<ILogicalExpression>> dedupExprs = new ArrayList<>(numPK);
for (LogicalVariable pkVar : newPkVars) {
@@ -471,9 +493,28 @@ public class VectorIndexAccessMethod implements IAccessMethod {
dedupOp.getInputs().add(new MutableObject<>(secondaryIndexUnnestOp));
dedupOp.setExecutionMode(secondaryIndexUnnestOp.getExecutionMode());
context.computeAndSetTypeEnvironmentForOperator(dedupOp);
- return dedupOp;
+ indexOnlyPlan = dedupOp;
}
- return secondaryIndexUnnestOp;
+
+ // The rewritten operators above now read variables only the search produces, so the search
+ // must be in the tree before any of them is retyped. Typed while the scan it replaces is still
+ // there, those variables have no producer: a bare reference merely types as unknown, but a
+ // function over one of them -- `m.year + 1`, `lower(m.title)`, a nested access on an INCLUDE
+ // column -- has no input type to compute from and fails. The caller replaces this same
+ // reference with what is returned here, which is harmless.
+ subTree.getDataSourceRef().setValue(indexOnlyPlan);
+
+ // Recompute type env bottom-up over every operator whose schema/expressions changed:
+ // the new secondary UnnestMap, the LIMIT chain, then each above-LIMIT ancestor from the
+ // one closest to LIMIT up to the root (aboveLimitOps is ordered root-first).
+ context.computeAndSetTypeEnvironmentForOperator(secondaryIndexUnnestOp);
+ org.apache.hyracks.algebricks.core.algebra.util.OperatorPropertiesUtil.typeOpRec(limitRef, context);
+ if (aboveLimitOps != null) {
+ for (int ai = aboveLimitOps.size() - 1; ai >= 0; ai--) {
+ context.computeAndSetTypeEnvironmentForOperator(aboveLimitOps.get(ai));
+ }
+ }
+ return indexOnlyPlan;
}
}
@@ -656,76 +697,6 @@ public class VectorIndexAccessMethod implements IAccessMethod {
return vectorDetails.getVectorParameters().getSimilarity();
}
- /**
- * Ensures an integer constant expression uses AInt32 instead of AInt64.
- *
- * SQL++ parser creates AInt64 (8 bytes) for all integer literals, but the runtime
- * uses IntegerPointable.getInteger() which reads only 4 bytes. Reading the first
- * 4 bytes of an 8-byte big-endian AInt64 for small values yields 0.
- *
- * This method converts AInt64 constants to AInt32 at compile time so the runtime
- * can correctly read them with IntegerPointable.
- *
- * @param expr The expression to check and potentially convert
- * @return The original expression if not an AInt64 constant, or a new AInt32 constant expression
- */
-
- /**
- * Index-only support: walk {@code op} and its descendants and replace every
- * {@code field-access-by-name(VarRef(recVar), "fieldName")} where {@code fieldName} is a PK column
- * with a direct {@code VarRef(pkVar)}. Used to repair ASSIGNs like
- * {@code $$id := field-access($$rec, "id")} after the DataSourceScan has been replaced by the
- * secondary UnnestMap (which no longer produces the record variable).
- */
- private static void rewriteRecordFieldAccessToPk(ILogicalOperator op, LogicalVariable recVar,
- Map<String, LogicalVariable> pkFieldToVar) {
- if (op == null) {
- return;
- }
- if (op.getOperatorTag() == LogicalOperatorTag.ASSIGN) {
- AssignOperator a = (AssignOperator) op;
- for (Mutable<ILogicalExpression> exprRef : a.getExpressions()) {
- rewriteRecordFieldAccessExpr(exprRef, recVar, pkFieldToVar);
- }
- }
- // Other operator types could in principle hold expressions referencing $$rec (e.g. SELECT,
- // ORDER, ASSIGNs further down). Walk all inputs and apply uniformly.
- for (Mutable<ILogicalOperator> input : op.getInputs()) {
- rewriteRecordFieldAccessToPk(input.getValue(), recVar, pkFieldToVar);
- }
- }
-
- private static void rewriteRecordFieldAccessExpr(Mutable<ILogicalExpression> exprRef, LogicalVariable recVar,
- Map<String, LogicalVariable> pkFieldToVar) {
- ILogicalExpression e = exprRef.getValue();
- if (e.getExpressionTag() != LogicalExpressionTag.FUNCTION_CALL) {
- return;
- }
- AbstractFunctionCallExpression fce = (AbstractFunctionCallExpression) e;
- if (BuiltinFunctions.FIELD_ACCESS_BY_NAME.equals(fce.getFunctionIdentifier())
- && fce.getArguments().size() == 2) {
- ILogicalExpression a0 = fce.getArguments().get(0).getValue();
- ILogicalExpression a1 = fce.getArguments().get(1).getValue();
- if (a0.getExpressionTag() == LogicalExpressionTag.VARIABLE
- && a1.getExpressionTag() == LogicalExpressionTag.CONSTANT) {
- LogicalVariable v0 = ((VariableReferenceExpression) a0).getVariableReference();
- if (recVar.equals(v0)) {
- String fieldName = extractStringFromConstant(a1);
- if (fieldName != null && pkFieldToVar.containsKey(fieldName)) {
- VariableReferenceExpression pkRef =
- new VariableReferenceExpression(pkFieldToVar.get(fieldName));
- pkRef.setSourceLocation(e.getSourceLocation());
- exprRef.setValue(pkRef);
- return;
- }
- }
- }
- }
- for (Mutable<ILogicalExpression> arg : fce.getArguments()) {
- rewriteRecordFieldAccessExpr(arg, recVar, pkFieldToVar);
- }
- }
-
/**
* Bind a {@code WHERE} over the index's {@code INCLUDE} columns to the vector search of an index-only
* plan: declare the columns it reads as outputs of the index-search unnest-map and rewrite the
@@ -740,37 +711,73 @@ public class VectorIndexAccessMethod implements IAccessMethod {
* unsupported plan shape — fail loudly instead of emitting a plan that silently drops every row.
*/
@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.GENERATED, notes = "Index-only ANN plans bind their INCLUDE-field predicate")
- private static void bindIncludeFilterToSearch(UnnestMapOperator unnestMap, SelectOperator selectOp,
- Index chosenIndex, Dataset dataset, ARecordType recordType, AbstractDataSourceOperator dataSourceOp,
- IOptimizationContext context) throws AlgebricksException {
+ private static void bindIncludeFilterToSearch(SelectOperator selectOp, Index chosenIndex, Dataset dataset,
+ ARecordType recordType, AbstractDataSourceOperator dataSourceOp,
+ VectorIncludeFilterPushdown.IncludeColumns includeColumns, IOptimizationContext context)
+ throws AlgebricksException {
if (selectOp == null) {
return;
}
- Index.VectorIndexDetails details = (Index.VectorIndexDetails) chosenIndex.getIndexDetails();
- int numPK = dataset.getPrimaryKeys().size();
- List<LogicalVariable> dsVars = dataSourceOp.getVariables();
- // Everything after the primary keys is the dataset record and, when present, the meta record: the
- // only bases a pushable field access may be rooted at.
- Set<LogicalVariable> recordVars = new HashSet<>(dsVars.subList(Math.min(numPK, dsVars.size()), dsVars.size()));
-
- VectorIncludeFilterPushdown.IndexContext idx =
- new VectorIncludeFilterPushdown.IndexContext(details.getIncludeFieldNames(), recordType,
- details.getVectorParameters().isQuantized(), numPK, recordVars);
- VectorIncludeFilterPushdown.PushedIncludeFilter pushed = VectorIncludeFilterPushdown
- .analyze(selectOp.getCondition().getValue(), selectOp, idx, context, context::newVar);
- if (pushed == null) {
+ ILogicalExpression bound = VectorIncludeFilterPushdown.bindPredicate(selectOp.getCondition().getValue(),
+ selectOp, includeContext(chosenIndex, dataset, recordType, dataSourceOp), context, includeColumns);
+ if (bound == null) {
throw new CompilationException(ErrorCode.COMPILATION_ILLEGAL_STATE,
"the vector index-only plan was chosen for a WHERE that cannot be pushed into the "
+ "INCLUDE fields of index " + chosenIndex.getIndexName());
}
- VectorIncludeFilterPushdown.declareFilterVariables(unnestMap, pushed);
// Rebind the SELECT to the INCLUDE columns instead of moving the predicate into the unnest-map's
// select condition. The record variable it used to read is about to be neutralized, so it cannot
// stay as it is -- but once it reads variables the unnest-map produces it is an ordinary predicate
// over ordinary variables, which every rule between here and the physical rewrites can handle.
// PushFilterIntoVectorSearchRule moves it into the select condition at the same point it does for
- // the lookup-and-rerank plan.
- selectOp.getCondition().setValue(pushed.condition());
+ // the lookup-and-rerank plan, and fails the compilation if by then it cannot; a predicate the rules
+ // in between prove true is simply removed, and nothing has to arrive.
+ selectOp.getCondition().setValue(bound);
+ }
+
+ /**
+ * The index-side inputs {@link VectorIncludeFilterPushdown} needs, with the record variable(s) a field
+ * access must be rooted at to be a candidate.
+ */
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_FABLE_5_1, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.ASSISTED, notes = "Root pushdown at the data record only, never the meta record")
+ private static VectorIncludeFilterPushdown.IndexContext includeContext(Index chosenIndex, Dataset dataset,
+ ARecordType recordType, AbstractDataSourceOperator dataSourceOp) {
+ int numPK = dataset.getPrimaryKeys().size();
+ List<LogicalVariable> dsVars = dataSourceOp.getVariables();
+ // The scan produces [pk..., record, meta?]. Only the record is a base: INCLUDE paths are resolved
+ // against the record type, so admitting the meta variable would let `meta(m).year` bind to a record
+ // column of the same name. IntroduceTopKAccessMethodRule's gate makes the same choice, and must: its
+ // verdict is what admits a predicate to the binding done here.
+ Set<LogicalVariable> recordVars = numPK < dsVars.size() ? Set.of(dsVars.get(numPK)) : Set.of();
+ return includeContext(chosenIndex, dataset, recordType, recordVars);
+ }
+
+ private static VectorIncludeFilterPushdown.IndexContext includeContext(Index chosenIndex, Dataset dataset,
+ ARecordType recordType, Set<LogicalVariable> recordVars) {
+ Index.VectorIndexDetails details = (Index.VectorIndexDetails) chosenIndex.getIndexDetails();
+ return new VectorIncludeFilterPushdown.IndexContext(details.getIncludeFieldNames(), recordType,
+ details.getVectorParameters().isQuantized(), dataset.getPrimaryKeys().size(), recordVars);
+ }
+
+ /**
+ * Point every reference to one of {@code pathToVar}'s paths, in {@code op} and its descendants, at the
+ * search output carrying that column, so the plan above no longer needs the assembled record.
+ */
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.GENERATED)
+ private static void bindPathsInDescendants(ILogicalOperator op, VectorIncludeFilterPushdown.IndexContext idx,
+ Map<List<String>, LogicalVariable> pathToVar, Map<LogicalVariable, ILogicalExpression> bindings)
+ throws AlgebricksException {
+ if (op == null || pathToVar == null || pathToVar.isEmpty()) {
+ return;
+ }
+ if (op.getOperatorTag() == LogicalOperatorTag.ASSIGN) {
+ for (Mutable<ILogicalExpression> exprRef : ((AssignOperator) op).getExpressions()) {
+ VectorIncludeFilterPushdown.bindPaths(exprRef, idx, pathToVar, bindings);
+ }
+ }
+ for (Mutable<ILogicalOperator> input : op.getInputs()) {
+ bindPathsInDescendants(input.getValue(), idx, pathToVar, bindings);
+ }
}
/**
@@ -783,7 +790,7 @@ public class VectorIndexAccessMethod implements IAccessMethod {
* on these values, but the type system still walks them.
*/
private static void neutralizeDanglingExpressions(ILogicalOperator root, List<LogicalVariable> oldRecordVars) {
- Set<LogicalVariable> dead = new java.util.HashSet<>(oldRecordVars);
+ Set<LogicalVariable> dead = new HashSet<>(oldRecordVars);
// Iterate to a fixpoint: each pass may neutralize an ASSIGN whose variable then feeds the next.
while (neutralizeDeadPass(root, dead)) {
// keep going until no new variable becomes dead
@@ -834,18 +841,4 @@ public class VectorIndexAccessMethod implements IAccessMethod {
}
return false;
}
-
- private static String extractStringFromConstant(ILogicalExpression e) {
- if (e.getExpressionTag() != LogicalExpressionTag.CONSTANT) {
- return null;
- }
- ConstantExpression ce = (ConstantExpression) e;
- IAlgebricksConstantValue v = ce.getValue();
- if (!(v instanceof AsterixConstantValue)) {
- return null;
- }
- IAObject obj = ((AsterixConstantValue) v).getObject();
- return (obj instanceof AString) ? ((AString) obj).getStringValue() : null;
- }
-
}
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml
index 88dcfb06da..babbaa7b1d 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/VectorQueries.xml
@@ -63,6 +63,16 @@
<output-dir compare="Clean-JSON">create-index-vtree-include-nested</output-dir>
</compilation-unit>
</test-case>
+ <test-case FilePath="vector">
+ <compilation-unit name="create-index-vtree-include-open-nested">
+ <output-dir compare="Clean-JSON">create-index-vtree-include-open-nested</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="vector">
+ <compilation-unit name="create-index-vtree-meta-include">
+ <output-dir compare="Clean-JSON">create-index-vtree-meta-include</output-dir>
+ </compilation-unit>
+ </test-case>
<test-case FilePath="vector">
<compilation-unit name="create-index-vtree-similarity-normalize">
<output-dir compare="Clean-JSON">create-index-vtree-similarity-normalize</output-dir>
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.15.query.sqlpp
similarity index 76%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.15.query.sqlpp
index 04afba50e2..4cceefae1f 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.15.query.sqlpp
@@ -18,10 +18,10 @@
*/
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+// A function over an INCLUDE column in the projection only. Index-only: `title` comes off the search and
+// upper() is applied to that variable. Filtered set is ids 6..30; nearest 3 are 6, 7, 8.
+SELECT m.id, upper(m.title) AS t
+FROM MovieMulti m
+WHERE m.year > 2005
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.16.plans.sqlpp
similarity index 65%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.16.plans.sqlpp
index 04afba50e2..6df3c29c1c 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.16.plans.sqlpp
@@ -17,11 +17,15 @@
* under the License.
*/
+-- param compile-only:string=true
+-- param optimized-logical-plan:string=true
+// CBO cost estimates vary with the cluster shape (partition count / core count), which
+// differs between a local run and CI. Disable the CBO so the plan is stable everywhere.
+SET `compiler.cbo` "false";
+// The .15 query as a plan assertion: index-only, upper() over the emitted `title` variable, no BTREE_SEARCH.
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+SELECT m.id, upper(m.title) AS t
+FROM MovieMulti m
+WHERE m.year > 2005
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.11.plans.sqlpp
similarity index 70%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.11.plans.sqlpp
index 04afba50e2..8ca51637dc 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.11.plans.sqlpp
@@ -17,11 +17,15 @@
* under the License.
*/
+
+-- param compile-only:string=true
+-- param optimized-logical-plan:string=true
+SET `compiler.cbo` "false";
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+// A predicate that reads no dataset field at all. Its variable set is empty, so every "is this variable
+// safe" test passes vacuously; only asking whether the predicate can actually be bound catches it.
+SELECT VALUE m.id
+FROM MovieSmall m
+WHERE random() > 0.5
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.16.query.sqlpp
similarity index 72%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.16.query.sqlpp
index 04afba50e2..8635526502 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.16.query.sqlpp
@@ -18,10 +18,11 @@
*/
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+// A function over the INCLUDE column, in both the WHERE and the projection. Index-only: the search emits
+// `year`, the predicate is evaluated in the cursor and the projection computes from the same variable.
+// Filtered set is ids 6..30; nearest 3 are 6, 7, 8.
+SELECT m.id, m.year + 1 AS y
+FROM MovieSmall m
+WHERE m.year + 1 > 2006
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.17.plans.sqlpp
similarity index 62%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.17.plans.sqlpp
index 04afba50e2..2ec1a52400 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.17.plans.sqlpp
@@ -17,11 +17,16 @@
* under the License.
*/
+-- param compile-only:string=true
+-- param optimized-logical-plan:string=true
+// CBO cost estimates vary with the cluster shape (partition count / core count), which
+// differs between a local run and CI. Disable the CBO so the plan is stable everywhere.
+SET `compiler.cbo` "false";
+// The .16 query as a plan assertion: index-only, with numeric-add over the emitted `year` variable in
+// both the pushed condition and the record constructor, and no BTREE_SEARCH.
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+SELECT m.id, m.year + 1 AS y
+FROM MovieSmall m
+WHERE m.year + 1 > 2006
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.18.query.sqlpp
similarity index 71%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.18.query.sqlpp
index 04afba50e2..5340d49ba7 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.18.query.sqlpp
@@ -18,10 +18,11 @@
*/
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+// A WHERE the optimizer proves true: `year` is declared, so IS NOT MISSING simplifies away and the SELECT is
+// removed before the physical rewrites. The index-only plan must not insist that a condition still reach the
+// search. Nearest 3 to the origin are ids 1, 2, 3.
+SELECT VALUE m.id
+FROM MovieSmall m
+WHERE m.year IS NOT MISSING
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.19.plans.sqlpp
similarity index 63%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.19.plans.sqlpp
index 04afba50e2..1aa44281e6 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.19.plans.sqlpp
@@ -17,11 +17,16 @@
* under the License.
*/
+-- param compile-only:string=true
+-- param optimized-logical-plan:string=true
+// CBO cost estimates vary with the cluster shape (partition count / core count), which
+// differs between a local run and CI. Disable the CBO so the plan is stable everywhere.
+SET `compiler.cbo` "false";
+// The .18 query as a plan assertion: index-only, no "condition (...)" on the vector unnest-map and no
+// SELECT anywhere, since the predicate was proven true.
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+SELECT VALUE m.id
+FROM MovieSmall m
+WHERE m.year IS NOT MISSING
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.8.query.sqlpp
similarity index 77%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.8.query.sqlpp
index 04afba50e2..e728b30dea 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.8.query.sqlpp
@@ -16,12 +16,12 @@
* specific language governing permissions and limitations
* under the License.
*/
-
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+
+// Returning an INCLUDE column, not just filtering on it. The value is in the secondary tuple the
+// search already reads, so this must be served by the index and must return the stored year.
+SELECT m.id, m.year
+FROM MovieSmall m
+WHERE m.year > 2005
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.9.plans.sqlpp
similarity index 62%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.9.plans.sqlpp
index 04afba50e2..ed3c6bfe42 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.9.plans.sqlpp
@@ -17,11 +17,20 @@
* under the License.
*/
+
+-- param compile-only:string=true
+-- param optimized-logical-plan:string=true
+
+// CBO cost estimates vary with the cluster shape, so disable it to keep the plan stable.
+SET `compiler.cbo` "false";
+
+// The .8 query as a plan assertion. `year` is both filtered on and returned, so it must be ONE
+// declared output of the vector search serving both -- the unnest-map carries [pk, dist, year], not
+// a second copy -- and there must be no BTREE_SEARCH above it.
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+
+SELECT m.id, m.year
+FROM MovieSmall m
+WHERE m.year > 2005
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-nested/create-index-vtree-include-nested.7.plans.sqlpp
similarity index 65%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-nested/create-index-vtree-include-nested.7.plans.sqlpp
index 04afba50e2..d549287edc 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-nested/create-index-vtree-include-nested.7.plans.sqlpp
@@ -17,11 +17,16 @@
* under the License.
*/
+
+-- param compile-only:string=true
+-- param optimized-logical-plan:string=true
+SET `compiler.cbo` "false";
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+// A NESTED INCLUDE column, filtered on and returned. Addressed by its full path [info, year], and served
+// index-only: the projection reads the column off the search even though the compiler splits the nested
+// access across two ASSIGNs, and the predicate is pushed into the search.
+SELECT m.id, m.info.year AS year
+FROM MovieSmall m
+WHERE m.info.year > 2005
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-nested/create-index-vtree-include-nested.8.plans.sqlpp
similarity index 59%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-nested/create-index-vtree-include-nested.8.plans.sqlpp
index 04afba50e2..970396aa55 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-nested/create-index-vtree-include-nested.8.plans.sqlpp
@@ -17,11 +17,17 @@
* under the License.
*/
+-- param compile-only:string=true
+-- param optimized-logical-plan:string=true
+// CBO cost estimates vary with the cluster shape (partition count / core count), which
+// differs between a local run and CI. Disable the CBO so the plan is stable everywhere.
+SET `compiler.cbo` "false";
+// The .6 query as a plan assertion: a WHERE on a NESTED INCLUDE column with a PK-only projection is
+// index-only, the predicate bound by its full path [info, year] into the vector unnest-map's "condition (...)",
+// and there is no BTREE_SEARCH.
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+SELECT VALUE m.id
+FROM MovieSmall m
+WHERE m.info.year > 2005
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.1.ddl.sqlpp
similarity index 64%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.1.ddl.sqlpp
index 04afba50e2..ffe6e0b968 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.1.ddl.sqlpp
@@ -17,11 +17,14 @@
* under the License.
*/
+// An INCLUDE column that is an undeclared (open) field holding a nested record. A query reading a
+// field inside it (m.info.year), in a projection or a predicate, is served by the column: the access
+// binds to the emitted variable and the remaining step is applied to it.
+DROP DATAVERSE test IF EXISTS;
+CREATE DATAVERSE test;
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
-LIMIT 3;
+CREATE TYPE MovieType AS OPEN {
+ id: int,
+ embedding: [double]
+};
+CREATE DATASET MovieNested(MovieType) PRIMARY KEY id;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.2.update.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.2.update.sqlpp
new file mode 100644
index 0000000000..ec62d9ce65
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.2.update.sqlpp
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+USE test;
+INSERT INTO MovieNested ([
+ {"id": 1, "info": {"year": 2001, "title": "t1"}, "embedding": [0.0, 0.0, 0.0, 0.1]},
+ {"id": 2, "info": {"year": 2002, "title": "t2"}, "embedding": [0.0, 0.0, 0.0, 0.2]},
+ {"id": 3, "info": {"year": 2003, "title": "t3"}, "embedding": [0.0, 0.0, 0.0, 0.3]},
+ {"id": 4, "info": {"year": 2004, "title": "t4"}, "embedding": [0.0, 0.0, 0.0, 0.4]},
+ {"id": 5, "info": {"year": 2005, "title": "t5"}, "embedding": [0.0, 0.0, 0.0, 0.5]},
+ {"id": 6, "info": {"year": 2006, "title": "t6"}, "embedding": [0.0, 0.0, 0.0, 0.6]},
+ {"id": 7, "info": {"year": 2007, "title": "t7"}, "embedding": [0.0, 0.0, 0.0, 0.7]},
+ {"id": 8, "info": {"year": 2008, "title": "t8"}, "embedding": [0.0, 0.0, 0.0, 0.8]},
+ {"id": 9, "info": {"year": 2009, "title": "t9"}, "embedding": [0.0, 0.0, 0.0, 0.9]},
+ {"id": 10, "info": {"year": 2010, "title": "t10"}, "embedding": [0.0, 0.0, 0.0, 1.0]},
+ {"id": 11, "info": {"year": 2011, "title": "t11"}, "embedding": [0.0, 0.0, 0.0, 1.1]},
+ {"id": 12, "info": {"year": 2012, "title": "t12"}, "embedding": [0.0, 0.0, 0.0, 1.2]},
+ {"id": 13, "info": {"year": 2013, "title": "t13"}, "embedding": [0.0, 0.0, 0.0, 1.3]},
+ {"id": 14, "info": {"year": 2014, "title": "t14"}, "embedding": [0.0, 0.0, 0.0, 1.4]},
+ {"id": 15, "info": {"year": 2015, "title": "t15"}, "embedding": [0.0, 0.0, 0.0, 1.5]},
+ {"id": 16, "info": {"year": 2016, "title": "t16"}, "embedding": [10.0, 10.0, 10.0, 0.1]},
+ {"id": 17, "info": {"year": 2017, "title": "t17"}, "embedding": [10.0, 10.0, 10.0, 0.2]},
+ {"id": 18, "info": {"year": 2018, "title": "t18"}, "embedding": [10.0, 10.0, 10.0, 0.3]},
+ {"id": 19, "info": {"year": 2019, "title": "t19"}, "embedding": [10.0, 10.0, 10.0, 0.4]},
+ {"id": 20, "info": {"year": 2020, "title": "t20"}, "embedding": [10.0, 10.0, 10.0, 0.5]},
+ {"id": 21, "info": {"year": 2021, "title": "t21"}, "embedding": [10.0, 10.0, 10.0, 0.6]},
+ {"id": 22, "info": {"year": 2022, "title": "t22"}, "embedding": [10.0, 10.0, 10.0, 0.7]},
+ {"id": 23, "info": {"year": 2023, "title": "t23"}, "embedding": [10.0, 10.0, 10.0, 0.8]},
+ {"id": 24, "info": {"year": 2024, "title": "t24"}, "embedding": [10.0, 10.0, 10.0, 0.9]},
+ {"id": 25, "info": {"year": 2025, "title": "t25"}, "embedding": [10.0, 10.0, 10.0, 1.0]},
+ {"id": 26, "info": {"year": 2026, "title": "t26"}, "embedding": [10.0, 10.0, 10.0, 1.1]},
+ {"id": 27, "info": {"year": 2027, "title": "t27"}, "embedding": [10.0, 10.0, 10.0, 1.2]},
+ {"id": 28, "info": {"year": 2028, "title": "t28"}, "embedding": [10.0, 10.0, 10.0, 1.3]},
+ {"id": 29, "info": {"year": 2029, "title": "t29"}, "embedding": [10.0, 10.0, 10.0, 1.4]},
+ {"id": 30, "info": {"year": 2030, "title": "t30"}, "embedding": [10.0, 10.0, 10.0, 1.5]}
+]);
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.3.ddl.sqlpp
similarity index 71%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.3.ddl.sqlpp
index 04afba50e2..e72b71c60f 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.3.ddl.sqlpp
@@ -17,11 +17,5 @@
* under the License.
*/
-USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
-LIMIT 3;
+// A sample index is required before creating a vector index (quantization training scans it).
+ANALYZE DATASET test.MovieNested WITH {"sample-seed": 1000};
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.4.ddl.sqlpp
similarity index 72%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.4.ddl.sqlpp
index 04afba50e2..1fe5fd4e28 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.4.ddl.sqlpp
@@ -18,10 +18,8 @@
*/
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
-LIMIT 3;
+CREATE INDEX idx_emb
+ ON MovieNested(embedding VECTOR)
+ INCLUDE (info)
+ TYPE VTREE
+ WITH { "dimension": 4, "similarity": "euclidean", "num_clusters": 2, "train_list_fraction": 1.0, "seed": 42 };
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.5.query.sqlpp
similarity index 78%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.5.query.sqlpp
index 04afba50e2..5bc5d75fbd 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.5.query.sqlpp
@@ -18,10 +18,8 @@
*/
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+// Nearest 3 to the origin are ids 1, 2, 3; `info` comes off the search and `.year` is read from it.
+SELECT m.id, m.info.year AS y
+FROM MovieNested m
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.6.plans.sqlpp
similarity index 64%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.6.plans.sqlpp
index 04afba50e2..5b81cbfe6e 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.6.plans.sqlpp
@@ -17,11 +17,15 @@
* under the License.
*/
+-- param compile-only:string=true
+-- param optimized-logical-plan:string=true
+// CBO cost estimates vary with the cluster shape (partition count / core count), which
+// differs between a local run and CI. Disable the CBO so the plan is stable everywhere.
+SET `compiler.cbo` "false";
+// The .5 query as a plan assertion: index-only, with the emitted `info` variable and a field access on it
+// in the record constructor, and no BTREE_SEARCH.
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+SELECT m.id, m.info.year AS y
+FROM MovieNested m
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.7.query.sqlpp
similarity index 78%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.7.query.sqlpp
index 04afba50e2..23c5418799 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.7.query.sqlpp
@@ -18,10 +18,9 @@
*/
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+// The same nested field in a WHERE. Filtered set is ids 6..30; nearest 3 are 6, 7, 8.
+SELECT m.id, m.info.year AS y
+FROM MovieNested m
+WHERE m.info.year > 2005
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.8.plans.sqlpp
similarity index 57%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.8.plans.sqlpp
index 04afba50e2..8ab0ee5cb1 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.8.plans.sqlpp
@@ -17,11 +17,17 @@
* under the License.
*/
+-- param compile-only:string=true
+-- param optimized-logical-plan:string=true
+// CBO cost estimates vary with the cluster shape (partition count / core count), which
+// differs between a local run and CI. Disable the CBO so the plan is stable everywhere.
+SET `compiler.cbo` "false";
+// The .7 query as a plan assertion. The predicate reads m.info.year and the column is `info`, so it binds
+// through the covering prefix, as the projection does: index-only, with a field access on the emitted
+// `info` variable inside the pushed condition, and no BTREE_SEARCH.
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+SELECT m.id, m.info.year AS y
+FROM MovieNested m
+WHERE m.info.year > 2005
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.99.ddl.sqlpp
similarity index 71%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.99.ddl.sqlpp
index 04afba50e2..20dc6fde5e 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.99.ddl.sqlpp
@@ -17,11 +17,4 @@
* under the License.
*/
-USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
-LIMIT 3;
+DROP DATAVERSE test IF EXISTS;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.1.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.1.ddl.sqlpp
new file mode 100644
index 0000000000..26cc2720a6
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.1.ddl.sqlpp
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// The primary key comes from meta() and the record has a field of the SAME name, `id`, which is an
+// INCLUDE column of the vector index. Neither may be served from the other: a predicate on meta().id
+// is not evaluable by the search, and a projection of it binds to the primary key, not the column.
+DROP DATAVERSE test IF EXISTS;
+CREATE DATAVERSE test;
+USE test;
+CREATE TYPE VecType AS OPEN {
+ id: int,
+ embedding: [double]
+};
+CREATE TYPE VecMetaType AS {
+ id: int
+};
+CREATE DATASET VecMeta(VecType) WITH META(VecMetaType)
+PRIMARY KEY META().id;
+CREATE FEED VecMetaStream WITH {
+ "adapter-name" : "localfs",
+ "reader" : "localfs",
+ "parser" : "record-with-metadata",
+ "type-name" : "VecType",
+ "meta-type-name" : "VecMetaType",
+ "path" : "asterix_nc1://data/csv/vector-meta-pk.csv",
+ "format" : "csv",
+ "delimiter" : ",",
+ "record-format" : "adm",
+ "record-index" : "1",
+ "key-indexes" : "0",
+ "key-indicators" : "1",
+ "header" : "false"
+};
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.2.update.sqlpp
similarity index 72%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.2.update.sqlpp
index 04afba50e2..335fd803f0 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.2.update.sqlpp
@@ -18,10 +18,6 @@
*/
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
-LIMIT 3;
+SET `wait-for-completion-feed` "true";
+CONNECT FEED VecMetaStream TO DATASET VecMeta;
+START FEED VecMetaStream;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.3.ddl.sqlpp
similarity index 71%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.3.ddl.sqlpp
index 04afba50e2..d7eb3f70ad 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.3.ddl.sqlpp
@@ -17,11 +17,4 @@
* under the License.
*/
-USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
-LIMIT 3;
+ANALYZE DATASET test.VecMeta WITH {"sample-seed": 1000};
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.4.ddl.sqlpp
similarity index 72%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.4.ddl.sqlpp
index 04afba50e2..4fe80a8b1e 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.4.ddl.sqlpp
@@ -18,10 +18,8 @@
*/
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
-LIMIT 3;
+CREATE INDEX idx_emb
+ ON VecMeta(embedding VECTOR)
+ INCLUDE (id)
+ TYPE VTREE
+ WITH { "dimension": 4, "similarity": "euclidean", "num_clusters": 2, "train_list_fraction": 1.0, "seed": 42 };
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.5.query.sqlpp
similarity index 76%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.5.query.sqlpp
index 04afba50e2..e200adca14 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.5.query.sqlpp
@@ -18,10 +18,10 @@
*/
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+// Meta PK and the record's INCLUDE column, both returned, with a WHERE on the column. Index-only.
+// Records have id 1..11 and meta().id 901..911; filtered set is id 3..11, nearest 3 are 3, 4, 5.
+SELECT VALUE [meta(m).id, m.id]
+FROM VecMeta m
+WHERE m.id > 2
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.6.plans.sqlpp
similarity index 66%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.6.plans.sqlpp
index 04afba50e2..db11aae033 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.6.plans.sqlpp
@@ -17,11 +17,15 @@
* under the License.
*/
+-- param compile-only:string=true
+-- param optimized-logical-plan:string=true
+// CBO cost estimates vary with the cluster shape (partition count / core count), which
+// differs between a local run and CI. Disable the CBO so the plan is stable everywhere.
+SET `compiler.cbo` "false";
+// The .5 query as a plan assertion: index-only with the condition on the emitted `id` column.
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+SELECT VALUE [meta(m).id, m.id]
+FROM VecMeta m
+WHERE m.id > 2
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.7.query.sqlpp
similarity index 77%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.7.query.sqlpp
index 04afba50e2..b0da8a58d9 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.7.query.sqlpp
@@ -18,10 +18,10 @@
*/
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+// A WHERE on meta().id. It must not be evaluated against the record's `id` INCLUDE column: bound there it
+// would compare 1..11 with 902 and return nothing. Correct answer: 903, 904, 905.
+SELECT VALUE meta(m).id
+FROM VecMeta m
+WHERE meta(m).id > 902
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.8.plans.sqlpp
similarity index 60%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.8.plans.sqlpp
index 04afba50e2..7f8d6bceb4 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.8.plans.sqlpp
@@ -17,11 +17,16 @@
* under the License.
*/
+-- param compile-only:string=true
+-- param optimized-logical-plan:string=true
+// CBO cost estimates vary with the cluster shape (partition count / core count), which
+// differs between a local run and CI. Disable the CBO so the plan is stable everywhere.
+SET `compiler.cbo` "false";
+// The .7 query as a plan assertion: no vector index is offered for a predicate the search cannot evaluate.
+// The predicate is on the primary key, so the plan is a primary-index range search with no VECTOR_SEARCH.
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
+SELECT VALUE meta(m).id
+FROM VecMeta m
+WHERE meta(m).id > 902
+ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.99.ddl.sqlpp
similarity index 71%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.99.ddl.sqlpp
index 04afba50e2..20dc6fde5e 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.99.ddl.sqlpp
@@ -17,11 +17,4 @@
* under the License.
*/
-USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
-EXPLAIN
-SELECT VALUE m.year
-FROM Vec m
-ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
-LIMIT 3;
+DROP DATAVERSE test IF EXISTS;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
index 04afba50e2..ea42a8da06 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
@@ -20,8 +20,11 @@
USE test;
// Every cluster probed and thirty candidates collected per requested row: the search touches the
// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
+// `category` is deliberately NOT an INCLUDE column of idx_emb -- the fetch and rerank this case
+// prices is exactly what the index-only plan removes, so projecting `year` instead would tip the
+// choice back to the index (see the case below).
EXPLAIN
-SELECT VALUE m.year
+SELECT VALUE m.category
FROM Vec m
ORDER BY ann_distance(m.embedding, [0.0, 0.0, 0.0, 0.0], "euclidean", 1.0, 30)
LIMIT 3;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.9.query.sqlpp
similarity index 76%
copy from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
copy to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.9.query.sqlpp
index 04afba50e2..7849ff5bb1 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.6.query.sqlpp
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/vtree-scan-choice/vtree-scan-choice.9.query.sqlpp
@@ -17,9 +17,11 @@
* under the License.
*/
+
USE test;
-// Every cluster probed and thirty candidates collected per requested row: the search touches the
-// whole partition and then reranks ninety fetched rows, so the plain data scan wins.
+// The .6 case with the projected column changed to the index's INCLUDE column, and nothing else.
+// The search reads `year` out of the secondary tuple and returns it, so the ninety-row fetch and
+// rerank that made the plain scan win there does not happen and the index wins instead.
EXPLAIN
SELECT VALUE m.year
FROM Vec m
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.11.plan b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.11.plan
index c8a635dbda..f5d04b99f7 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.11.plan
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.11.plan
@@ -28,11 +28,15 @@ distribute result [$$19]
-- STABLE_SORT [$$35(ASC), $$36(ASC)] |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- unnest-map [$$35, $$36] <- index-search("idx_cpk_emb", 8, "Default", "testcpk", "MovieCpk", false, false, false, 5, $$30, $$31, $$32, $$33, $$34)
- -- VECTOR_SEARCH |PARTITIONED|
+ project ([$$35, $$36])
+ -- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- assign [$$30, $$31, $$32, $$33, $$34] <- [array: [ 0.0, 0.0, 0.0, 0.0 ], 3, "euclidean", 1.0, 1]
- -- ASSIGN |PARTITIONED|
- empty-tuple-source
- -- EMPTY_TUPLE_SOURCE |PARTITIONED|
+ unnest-map [$$35, $$36, $$37] <- index-search("idx_cpk_emb", 8, "Default", "testcpk", "MovieCpk", false, false, false, 5, $$30, $$31, $$32, $$33, $$34)
+ -- VECTOR_SEARCH |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ assign [$$30, $$31, $$32, $$33, $$34] <- [array: [ 0.0, 0.0, 0.0, 0.0 ], 3, "euclidean", 1.0, 1]
+ -- ASSIGN |PARTITIONED|
+ empty-tuple-source
+ -- EMPTY_TUPLE_SOURCE |PARTITIONED|
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.8.plan b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.8.plan
index fb08757c27..7272cba627 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.8.plan
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.8.plan
@@ -18,13 +18,15 @@ distribute result [$$18]
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
assign [$$18] <- [ordered-list-constructor($$33, $$34)] project: [$$18, $$35]
-- ASSIGN |PARTITIONED|
- exchange
- -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- unnest-map [$$33, $$34, $$35] <- index-search("idx_cpk_emb", 8, "Default", "testcpk", "MovieCpk", false, false, true, 5, $$28, $$29, $$30, $$31, $$32)
- -- VECTOR_SEARCH |PARTITIONED|
- exchange
- -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- assign [$$28, $$29, $$30, $$31, $$32] <- [array: [ 0.0, 0.0, 0.0, 0.0 ], 3, "euclidean", 1.0, 1]
- -- ASSIGN |PARTITIONED|
- empty-tuple-source
- -- EMPTY_TUPLE_SOURCE |PARTITIONED|
+ project ([$$33, $$34, $$35])
+ -- STREAM_PROJECT |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ unnest-map [$$33, $$34, $$35, $$36] <- index-search("idx_cpk_emb", 8, "Default", "testcpk", "MovieCpk", false, false, true, 5, $$28, $$29, $$30, $$31, $$32)
+ -- VECTOR_SEARCH |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ assign [$$28, $$29, $$30, $$31, $$32] <- [array: [ 0.0, 0.0, 0.0, 0.0 ], 3, "euclidean", 1.0, 1]
+ -- ASSIGN |PARTITIONED|
+ empty-tuple-source
+ -- EMPTY_TUPLE_SOURCE |PARTITIONED|
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.13.plan b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.13.plan
index 92081a3c99..56aba3b027 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.13.plan
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.13.plan
@@ -20,7 +20,7 @@ distribute result [$$33]
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- unnest-map [$$33, $$34, $$35] <- index-search("idx_emb", 8, "Default", "test", "MovieMulti", false, false, true, 5, $$28, $$29, $$30, $$31, $$32) condition (eq({"y": $$35}, { "y": 2006 }))
+ unnest-map [$$33, $$34, $$35, $$36, $$37, $$38] <- index-search("idx_emb", 8, "Default", "test", "MovieMulti", false, false, true, 5, $$28, $$29, $$30, $$31, $$32) condition (eq({"y": $$35}, { "y": 2006 }))
-- VECTOR_SEARCH |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.14.plan b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.14.plan
index 241fd98442..bbf74af938 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.14.plan
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.14.plan
@@ -20,7 +20,7 @@ distribute result [$$36]
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- unnest-map [$$36, $$37, $$38, $$39] <- index-search("idx_emb", 8, "Default", "test", "MovieMulti", false, false, true, 5, $$31, $$32, $$33, $$34, $$35) condition (and(gt(if-null($$38, 0), 2005), eq($$39, 5)))
+ unnest-map [$$36, $$37, $$38, $$39, $$40, $$41] <- index-search("idx_emb", 8, "Default", "test", "MovieMulti", false, false, true, 5, $$31, $$32, $$33, $$34, $$35) condition (and(gt(if-null($$38, 0), 2005), eq($$39, 5)))
-- VECTOR_SEARCH |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.15.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.15.adm
new file mode 100644
index 0000000000..f883b45bfa
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.15.adm
@@ -0,0 +1,3 @@
+{ "id": 6, "t": "T6" }
+{ "id": 7, "t": "T7" }
+{ "id": 8, "t": "T8" }
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.16.plan b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.16.plan
new file mode 100644
index 0000000000..ec4a70581e
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.16.plan
@@ -0,0 +1,32 @@
+distribute result [$$21]
+-- DISTRIBUTE_RESULT |UNPARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |UNPARTITIONED|
+ limit 3
+ -- STREAM_LIMIT |UNPARTITIONED|
+ project ([$$21])
+ -- STREAM_PROJECT |PARTITIONED|
+ exchange
+ -- SORT_MERGE_EXCHANGE [$$37(ASC) ] |PARTITIONED|
+ limit 3
+ -- STREAM_LIMIT |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ order (topK: 3) (ASC, $$37)
+ -- STABLE_SORT [topK: 3] [$$37(ASC)] |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ assign [$$21] <- [{"id": $$36, "t": uppercase($$40)}] project: [$$21, $$37]
+ -- ASSIGN |PARTITIONED|
+ project ([$$36, $$37, $$40])
+ -- STREAM_PROJECT |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ unnest-map [$$36, $$37, $$38, $$39, $$40, $$41] <- index-search("idx_emb", 8, "Default", "test", "MovieMulti", false, false, true, 5, $$31, $$32, $$33, $$34, $$35) condition (gt($$38, 2005))
+ -- VECTOR_SEARCH |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ assign [$$31, $$32, $$33, $$34, $$35] <- [array: [ 0.0, 0.0, 0.0, 0.0 ], 3, "euclidean", 1.0, 1]
+ -- ASSIGN |PARTITIONED|
+ empty-tuple-source
+ -- EMPTY_TUPLE_SOURCE |PARTITIONED|
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.7.plan b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.7.plan
index cb3862b9e0..3a62e0d87f 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.7.plan
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.7.plan
@@ -20,7 +20,7 @@ distribute result [$$35]
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- unnest-map [$$35, $$36, $$37, $$38] <- index-search("idx_emb", 8, "Default", "test", "MovieMulti", false, false, true, 5, $$30, $$31, $$32, $$33, $$34) condition (and(gt($$37, 2005), eq($$38, 5)))
+ unnest-map [$$35, $$36, $$37, $$38, $$39, $$40] <- index-search("idx_emb", 8, "Default", "test", "MovieMulti", false, false, true, 5, $$30, $$31, $$32, $$33, $$34) condition (and(gt($$37, 2005), eq($$38, 5)))
-- VECTOR_SEARCH |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.11.plan b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.11.plan
new file mode 100644
index 0000000000..f9fb022cee
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.11.plan
@@ -0,0 +1,30 @@
+distribute result [$$23]
+-- DISTRIBUTE_RESULT |UNPARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |UNPARTITIONED|
+ limit 3
+ -- STREAM_LIMIT |UNPARTITIONED|
+ project ([$$23])
+ -- STREAM_PROJECT |PARTITIONED|
+ exchange
+ -- SORT_MERGE_EXCHANGE [$$22(ASC) ] |PARTITIONED|
+ limit 3
+ -- STREAM_LIMIT |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ order (topK: 3) (ASC, $$22)
+ -- STABLE_SORT [topK: 3] [$$22(ASC)] |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ assign [$$22] <- [euclidean-distance($$m.getField(2), array: [ 0.0, 0.0, 0.0, 0.0 ])] project: [$$23, $$22]
+ -- ASSIGN |PARTITIONED|
+ select (gt(random(), 0.5))
+ -- STREAM_SELECT |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ data-scan []<-[$$23, $$m] <- test.MovieSmall
+ -- DATASOURCE_SCAN |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ empty-tuple-source
+ -- EMPTY_TUPLE_SOURCE |PARTITIONED|
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.16.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.16.adm
new file mode 100644
index 0000000000..56c314eebf
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.16.adm
@@ -0,0 +1,3 @@
+{ "id": 6, "y": 2007 }
+{ "id": 7, "y": 2008 }
+{ "id": 8, "y": 2009 }
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.17.plan b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.17.plan
new file mode 100644
index 0000000000..080fdf65fd
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.17.plan
@@ -0,0 +1,32 @@
+distribute result [$$22]
+-- DISTRIBUTE_RESULT |UNPARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |UNPARTITIONED|
+ limit 3
+ -- STREAM_LIMIT |UNPARTITIONED|
+ project ([$$22])
+ -- STREAM_PROJECT |PARTITIONED|
+ exchange
+ -- SORT_MERGE_EXCHANGE [$$40(ASC) ] |PARTITIONED|
+ limit 3
+ -- STREAM_LIMIT |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ order (topK: 3) (ASC, $$40)
+ -- STABLE_SORT [topK: 3] [$$40(ASC)] |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ assign [$$22] <- [{"id": $$39, "y": $$42}] project: [$$22, $$40]
+ -- ASSIGN |PARTITIONED|
+ assign [$$42] <- [numeric-add($$41, 1)] project: [$$39, $$40, $$42]
+ -- ASSIGN |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ unnest-map [$$39, $$40, $$41] <- index-search("idx_emb", 8, "Default", "test", "MovieSmall", false, false, true, 5, $$34, $$35, $$36, $$37, $$38) condition (gt(numeric-add($$41, 1), 2006))
+ -- VECTOR_SEARCH |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ assign [$$34, $$35, $$36, $$37, $$38] <- [array: [ 0.0, 0.0, 0.0, 0.0 ], 3, "euclidean", 1.0, 1]
+ -- ASSIGN |PARTITIONED|
+ empty-tuple-source
+ -- EMPTY_TUPLE_SOURCE |PARTITIONED|
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.18.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.18.adm
new file mode 100644
index 0000000000..01e79c32a8
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.18.adm
@@ -0,0 +1,3 @@
+1
+2
+3
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.13.plan b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.19.plan
similarity index 63%
copy from asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.13.plan
copy to asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.19.plan
index 92081a3c99..d68fa5b296 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.13.plan
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.19.plan
@@ -1,30 +1,30 @@
-distribute result [$$33]
+distribute result [$$32]
-- DISTRIBUTE_RESULT |UNPARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |UNPARTITIONED|
limit 3
-- STREAM_LIMIT |UNPARTITIONED|
- project ([$$33])
+ project ([$$32])
-- STREAM_PROJECT |PARTITIONED|
exchange
- -- SORT_MERGE_EXCHANGE [$$34(ASC) ] |PARTITIONED|
+ -- SORT_MERGE_EXCHANGE [$$33(ASC) ] |PARTITIONED|
limit 3
-- STREAM_LIMIT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- order (topK: 3) (ASC, $$34)
- -- STABLE_SORT [topK: 3] [$$34(ASC)] |PARTITIONED|
+ order (topK: 3) (ASC, $$33)
+ -- STABLE_SORT [topK: 3] [$$33(ASC)] |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- project ([$$33, $$34])
+ project ([$$32, $$33])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- unnest-map [$$33, $$34, $$35] <- index-search("idx_emb", 8, "Default", "test", "MovieMulti", false, false, true, 5, $$28, $$29, $$30, $$31, $$32) condition (eq({"y": $$35}, { "y": 2006 }))
+ unnest-map [$$32, $$33, $$34] <- index-search("idx_emb", 8, "Default", "test", "MovieSmall", false, false, true, 5, $$27, $$28, $$29, $$30, $$31)
-- VECTOR_SEARCH |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- assign [$$28, $$29, $$30, $$31, $$32] <- [array: [ 0.0, 0.0, 0.0, 0.0 ], 3, "euclidean", 1.0, 1]
+ assign [$$27, $$28, $$29, $$30, $$31] <- [array: [ 0.0, 0.0, 0.0, 0.0 ], 3, "euclidean", 1.0, 1]
-- ASSIGN |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.8.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.8.adm
new file mode 100644
index 0000000000..7442f049b2
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.8.adm
@@ -0,0 +1,3 @@
+{ "id": 6, "year": 2006 }
+{ "id": 7, "year": 2007 }
+{ "id": 8, "year": 2008 }
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.8.plan b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.9.plan
similarity index 73%
copy from asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.8.plan
copy to asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.9.plan
index fb08757c27..64141303d0 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.8.plan
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter/create-index-vtree-include-filter.9.plan
@@ -1,10 +1,10 @@
-distribute result [$$18]
+distribute result [$$20]
-- DISTRIBUTE_RESULT |UNPARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |UNPARTITIONED|
limit 3
-- STREAM_LIMIT |UNPARTITIONED|
- project ([$$18])
+ project ([$$20])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- SORT_MERGE_EXCHANGE [$$35(ASC) ] |PARTITIONED|
@@ -16,15 +16,15 @@ distribute result [$$18]
-- STABLE_SORT [topK: 3] [$$35(ASC)] |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- assign [$$18] <- [ordered-list-constructor($$33, $$34)] project: [$$18, $$35]
+ assign [$$20] <- [{"id": $$34, "year": $$36}] project: [$$20, $$35]
-- ASSIGN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- unnest-map [$$33, $$34, $$35] <- index-search("idx_cpk_emb", 8, "Default", "testcpk", "MovieCpk", false, false, true, 5, $$28, $$29, $$30, $$31, $$32)
+ unnest-map [$$34, $$35, $$36] <- index-search("idx_emb", 8, "Default", "test", "MovieSmall", false, false, true, 5, $$29, $$30, $$31, $$32, $$33) condition (gt($$36, 2005))
-- VECTOR_SEARCH |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- assign [$$28, $$29, $$30, $$31, $$32] <- [array: [ 0.0, 0.0, 0.0, 0.0 ], 3, "euclidean", 1.0, 1]
+ assign [$$29, $$30, $$31, $$32, $$33] <- [array: [ 0.0, 0.0, 0.0, 0.0 ], 3, "euclidean", 1.0, 1]
-- ASSIGN |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.8.plan b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-nested/create-index-vtree-include-nested.7.plan
similarity index 61%
copy from asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.8.plan
copy to asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-nested/create-index-vtree-include-nested.7.plan
index fb08757c27..78772c5dbd 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.8.plan
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-nested/create-index-vtree-include-nested.7.plan
@@ -1,30 +1,30 @@
-distribute result [$$18]
+distribute result [$$22]
-- DISTRIBUTE_RESULT |UNPARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |UNPARTITIONED|
limit 3
-- STREAM_LIMIT |UNPARTITIONED|
- project ([$$18])
+ project ([$$22])
-- STREAM_PROJECT |PARTITIONED|
exchange
- -- SORT_MERGE_EXCHANGE [$$35(ASC) ] |PARTITIONED|
+ -- SORT_MERGE_EXCHANGE [$$38(ASC) ] |PARTITIONED|
limit 3
-- STREAM_LIMIT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- order (topK: 3) (ASC, $$35)
- -- STABLE_SORT [topK: 3] [$$35(ASC)] |PARTITIONED|
+ order (topK: 3) (ASC, $$38)
+ -- STABLE_SORT [topK: 3] [$$38(ASC)] |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- assign [$$18] <- [ordered-list-constructor($$33, $$34)] project: [$$18, $$35]
+ assign [$$22] <- [{"id": $$37, "year": $$39}] project: [$$22, $$38]
-- ASSIGN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- unnest-map [$$33, $$34, $$35] <- index-search("idx_cpk_emb", 8, "Default", "testcpk", "MovieCpk", false, false, true, 5, $$28, $$29, $$30, $$31, $$32)
+ unnest-map [$$37, $$38, $$39] <- index-search("idx_emb", 8, "Default", "test", "MovieSmall", false, false, true, 5, $$32, $$33, $$34, $$35, $$36) condition (gt($$39, 2005))
-- VECTOR_SEARCH |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- assign [$$28, $$29, $$30, $$31, $$32] <- [array: [ 0.0, 0.0, 0.0, 0.0 ], 3, "euclidean", 1.0, 1]
+ assign [$$32, $$33, $$34, $$35, $$36] <- [array: [ 0.0, 0.0, 0.0, 0.0 ], 3, "euclidean", 1.0, 1]
-- ASSIGN |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.13.plan b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-nested/create-index-vtree-include-nested.8.plan
similarity index 89%
copy from asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.13.plan
copy to asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-nested/create-index-vtree-include-nested.8.plan
index 92081a3c99..c47e90ec69 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.13.plan
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-nested/create-index-vtree-include-nested.8.plan
@@ -20,7 +20,7 @@ distribute result [$$33]
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- unnest-map [$$33, $$34, $$35] <- index-search("idx_emb", 8, "Default", "test", "MovieMulti", false, false, true, 5, $$28, $$29, $$30, $$31, $$32) condition (eq({"y": $$35}, { "y": 2006 }))
+ unnest-map [$$33, $$34, $$35] <- index-search("idx_emb", 8, "Default", "test", "MovieSmall", false, false, true, 5, $$28, $$29, $$30, $$31, $$32) condition (gt($$35, 2005))
-- VECTOR_SEARCH |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.5.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.5.adm
new file mode 100644
index 0000000000..8d67718b60
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.5.adm
@@ -0,0 +1,3 @@
+{ "id": 1, "y": 2001 }
+{ "id": 2, "y": 2002 }
+{ "id": 3, "y": 2003 }
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.8.plan b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.6.plan
similarity index 73%
copy from asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.8.plan
copy to asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.6.plan
index fb08757c27..5dbded29c9 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-composite-pk/create-index-vtree-composite-pk.8.plan
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.6.plan
@@ -1,10 +1,10 @@
-distribute result [$$18]
+distribute result [$$19]
-- DISTRIBUTE_RESULT |UNPARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |UNPARTITIONED|
limit 3
-- STREAM_LIMIT |UNPARTITIONED|
- project ([$$18])
+ project ([$$19])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- SORT_MERGE_EXCHANGE [$$35(ASC) ] |PARTITIONED|
@@ -16,15 +16,15 @@ distribute result [$$18]
-- STABLE_SORT [topK: 3] [$$35(ASC)] |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- assign [$$18] <- [ordered-list-constructor($$33, $$34)] project: [$$18, $$35]
+ assign [$$19] <- [{"id": $$34, "y": $$36.getField("year")}] project: [$$19, $$35]
-- ASSIGN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- unnest-map [$$33, $$34, $$35] <- index-search("idx_cpk_emb", 8, "Default", "testcpk", "MovieCpk", false, false, true, 5, $$28, $$29, $$30, $$31, $$32)
+ unnest-map [$$34, $$35, $$36] <- index-search("idx_emb", 8, "Default", "test", "MovieNested", false, false, true, 5, $$29, $$30, $$31, $$32, $$33)
-- VECTOR_SEARCH |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- assign [$$28, $$29, $$30, $$31, $$32] <- [array: [ 0.0, 0.0, 0.0, 0.0 ], 3, "euclidean", 1.0, 1]
+ assign [$$29, $$30, $$31, $$32, $$33] <- [array: [ 0.0, 0.0, 0.0, 0.0 ], 3, "euclidean", 1.0, 1]
-- ASSIGN |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.7.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.7.adm
new file mode 100644
index 0000000000..83e1991a31
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.7.adm
@@ -0,0 +1,3 @@
+{ "id": 6, "y": 2006 }
+{ "id": 7, "y": 2007 }
+{ "id": 8, "y": 2008 }
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.8.plan b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.8.plan
new file mode 100644
index 0000000000..0bc24d6121
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-open-nested/create-index-vtree-include-open-nested.8.plan
@@ -0,0 +1,32 @@
+distribute result [$$22]
+-- DISTRIBUTE_RESULT |UNPARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |UNPARTITIONED|
+ limit 3
+ -- STREAM_LIMIT |UNPARTITIONED|
+ project ([$$22])
+ -- STREAM_PROJECT |PARTITIONED|
+ exchange
+ -- SORT_MERGE_EXCHANGE [$$38(ASC) ] |PARTITIONED|
+ limit 3
+ -- STREAM_LIMIT |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ order (topK: 3) (ASC, $$38)
+ -- STABLE_SORT [topK: 3] [$$38(ASC)] |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ assign [$$22] <- [{"id": $$37, "y": $$27}] project: [$$22, $$38]
+ -- ASSIGN |PARTITIONED|
+ assign [$$27] <- [$$39.getField("year")] project: [$$37, $$38, $$27]
+ -- ASSIGN |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ unnest-map [$$37, $$38, $$39] <- index-search("idx_emb", 8, "Default", "test", "MovieNested", false, false, true, 5, $$32, $$33, $$34, $$35, $$36) condition (gt($$39.getField("year"), 2005))
+ -- VECTOR_SEARCH |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ assign [$$32, $$33, $$34, $$35, $$36] <- [array: [ 0.0, 0.0, 0.0, 0.0 ], 3, "euclidean", 1.0, 1]
+ -- ASSIGN |PARTITIONED|
+ empty-tuple-source
+ -- EMPTY_TUPLE_SOURCE |PARTITIONED|
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.5.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.5.adm
new file mode 100644
index 0000000000..5315e2ccd7
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.5.adm
@@ -0,0 +1,3 @@
+[ 903, 3 ]
+[ 904, 4 ]
+[ 905, 5 ]
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.14.plan b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.6.plan
similarity index 74%
copy from asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.14.plan
copy to asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.6.plan
index 241fd98442..16a728ebc9 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-include-filter-multi/create-index-vtree-include-filter-multi.14.plan
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.6.plan
@@ -1,10 +1,10 @@
-distribute result [$$36]
+distribute result [$$21]
-- DISTRIBUTE_RESULT |UNPARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |UNPARTITIONED|
limit 3
-- STREAM_LIMIT |UNPARTITIONED|
- project ([$$36])
+ project ([$$21])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- SORT_MERGE_EXCHANGE [$$37(ASC) ] |PARTITIONED|
@@ -16,11 +16,11 @@ distribute result [$$36]
-- STABLE_SORT [topK: 3] [$$37(ASC)] |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- project ([$$36, $$37])
- -- STREAM_PROJECT |PARTITIONED|
+ assign [$$21] <- [ordered-list-constructor($$36, $$38)] project: [$$21, $$37]
+ -- ASSIGN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- unnest-map [$$36, $$37, $$38, $$39] <- index-search("idx_emb", 8, "Default", "test", "MovieMulti", false, false, true, 5, $$31, $$32, $$33, $$34, $$35) condition (and(gt(if-null($$38, 0), 2005), eq($$39, 5)))
+ unnest-map [$$36, $$37, $$38] <- index-search("idx_emb", 8, "Default", "test", "VecMeta", false, false, true, 5, $$31, $$32, $$33, $$34, $$35) condition (gt($$38, 2))
-- VECTOR_SEARCH |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.7.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.7.adm
new file mode 100644
index 0000000000..42e6cafa85
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.7.adm
@@ -0,0 +1,3 @@
+903
+904
+905
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.8.plan b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.8.plan
new file mode 100644
index 0000000000..74385ce068
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-meta-include/create-index-vtree-meta-include.8.plan
@@ -0,0 +1,32 @@
+distribute result [$$26]
+-- DISTRIBUTE_RESULT |UNPARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |UNPARTITIONED|
+ limit 3
+ -- STREAM_LIMIT |UNPARTITIONED|
+ project ([$$26])
+ -- STREAM_PROJECT |PARTITIONED|
+ exchange
+ -- SORT_MERGE_EXCHANGE [$$24(ASC) ] |PARTITIONED|
+ limit 3
+ -- STREAM_LIMIT |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ order (topK: 3) (ASC, $$24)
+ -- STABLE_SORT [topK: 3] [$$24(ASC)] |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ assign [$$24] <- [euclidean-distance($$m.getField(1), array: [ 0.0, 0.0, 0.0, 0.0 ])] project: [$$26, $$24]
+ -- ASSIGN |PARTITIONED|
+ project ([$$26, $$m])
+ -- STREAM_PROJECT |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ unnest-map [$$26, $$m, $$27] <- index-search("VecMeta", 0, "Default", "test", "VecMeta", false, false, 1, $$29, 0, false, true, false)
+ -- BTREE_SEARCH |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ assign [$$29] <- [902]
+ -- ASSIGN |PARTITIONED|
+ empty-tuple-source
+ -- EMPTY_TUPLE_SOURCE |PARTITIONED|
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-nested/create-index-vtree-nested.5.plan b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-nested/create-index-vtree-nested.5.plan
index 92d0bdb792..85003991f0 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-nested/create-index-vtree-nested.5.plan
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/create-index-vtree-nested/create-index-vtree-nested.5.plan
@@ -16,11 +16,15 @@ distribute result [$$30]
-- STABLE_SORT [topK: 3] [$$31(ASC)] |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- unnest-map [$$30, $$31] <- index-search("idx_emb", 8, "Default", "test", "MovieNested", false, false, true, 5, $$25, $$26, $$27, $$28, $$29)
- -- VECTOR_SEARCH |PARTITIONED|
+ project ([$$30, $$31])
+ -- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
- assign [$$25, $$26, $$27, $$28, $$29] <- [array: [ 0.0, 0.0, 0.0, 0.0 ], 3, "euclidean", 1.0, 1]
- -- ASSIGN |PARTITIONED|
- empty-tuple-source
- -- EMPTY_TUPLE_SOURCE |PARTITIONED|
+ unnest-map [$$30, $$31, $$32] <- index-search("idx_emb", 8, "Default", "test", "MovieNested", false, false, true, 5, $$25, $$26, $$27, $$28, $$29)
+ -- VECTOR_SEARCH |PARTITIONED|
+ exchange
+ -- ONE_TO_ONE_EXCHANGE |PARTITIONED|
+ assign [$$25, $$26, $$27, $$28, $$29] <- [array: [ 0.0, 0.0, 0.0, 0.0 ], 3, "euclidean", 1.0, 1]
+ -- ASSIGN |PARTITIONED|
+ empty-tuple-source
+ -- EMPTY_TUPLE_SOURCE |PARTITIONED|
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/vtree-scan-choice/vtree-scan-choice.5.regex b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/vtree-scan-choice/vtree-scan-choice.5.regex
new file mode 100644
index 0000000000..c276abede7
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/vector/vtree-scan-choice/vtree-scan-choice.5.regex
@@ -0,0 +1,4 @@
+/-- VECTOR_SEARCH \|PARTITIONED\|/
+/index-search\("idx_emb", /
+-/data-scan \[\]<-\[.*\] <- test\.Vec/
+-/BTREE_SEARCH/