You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Stefanietry (via GitHub)" <gi...@apache.org> on 2026/06/16 08:03:58 UTC
[PR] [spark] support lateral inner join for vector search [paimon]
Stefanietry opened a new pull request, #8252:
URL: https://github.com/apache/paimon/pull/8252
Purpose
Purpose: Support lateral join for vector search on spark.
Linked issue: https://github.com/apache/paimon/issues/8251
Tests
Add vector search with lateral join on org.apache.paimon.spark.SparkMultimodalITCase#testVectorăorg.apache.spark.sql.test.SQLTestUtils#test("lateral vector search preserves subquery alias qualifiers")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] support lateral inner join for vector search [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8252:
URL: https://github.com/apache/paimon/pull/8252#discussion_r3419164239
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/execution/PaimonStrategy.scala:
##########
@@ -215,3 +232,133 @@ case class PaimonStrategy(spark: SparkSession)
SparkShimLoader.shim.classicApi.recacheByPlan(spark, v2Relation)
}
}
+
+case class LateralVectorSearchExec(
+ innerTable: InnerTable,
+ columnName: String,
+ queryVectorExpr: Expression,
+ limit: Int,
+ vectorSearchOutput: Seq[Attribute],
+ projectList: Seq[NamedExpression],
+ projectOutput: Seq[Attribute],
+ child: SparkPlan)
+ extends SparkPlan {
+
+ override def children: Seq[SparkPlan] = Seq(child)
+
+ override def output: Seq[Attribute] = child.output ++ projectOutput
+
+ @transient override lazy val producedAttributes: AttributeSet = AttributeSet(vectorSearchOutput)
+
+ @transient
+ override lazy val references: AttributeSet = {
+ AttributeSet.fromAttributeSets(expressions.map(_.references)) -- producedAttributes
+ }
+
+ override protected def withNewChildrenInternal(newChildren: IndexedSeq[SparkPlan]): SparkPlan = {
+ copy(child = newChildren.head)
+ }
+
+ override protected def doExecute(): RDD[InternalRow] = {
+ child.execute().mapPartitions {
+ outerRows =>
Review Comment:
Can batch queries be supported? Batch queries are crucial for performance. You can take a look to benchmark in https://github.com/apache/paimon-vector-index
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] support lateral inner join for vector search [paimon]
Posted by "Stefanietry (via GitHub)" <gi...@apache.org>.
Stefanietry commented on code in PR #8252:
URL: https://github.com/apache/paimon/pull/8252#discussion_r3419478316
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/execution/PaimonStrategy.scala:
##########
@@ -215,3 +232,133 @@ case class PaimonStrategy(spark: SparkSession)
SparkShimLoader.shim.classicApi.recacheByPlan(spark, v2Relation)
}
}
+
+case class LateralVectorSearchExec(
+ innerTable: InnerTable,
+ columnName: String,
+ queryVectorExpr: Expression,
+ limit: Int,
+ vectorSearchOutput: Seq[Attribute],
+ projectList: Seq[NamedExpression],
+ projectOutput: Seq[Attribute],
+ child: SparkPlan)
+ extends SparkPlan {
+
+ override def children: Seq[SparkPlan] = Seq(child)
+
+ override def output: Seq[Attribute] = child.output ++ projectOutput
+
+ @transient override lazy val producedAttributes: AttributeSet = AttributeSet(vectorSearchOutput)
+
+ @transient
+ override lazy val references: AttributeSet = {
+ AttributeSet.fromAttributeSets(expressions.map(_.references)) -- producedAttributes
+ }
+
+ override protected def withNewChildrenInternal(newChildren: IndexedSeq[SparkPlan]): SparkPlan = {
+ copy(child = newChildren.head)
+ }
+
+ override protected def doExecute(): RDD[InternalRow] = {
+ child.execute().mapPartitions {
+ outerRows =>
Review Comment:
Thanks for your reminder. I'll refine it in batch mode later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] support lateral inner join for vector search [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8252:
URL: https://github.com/apache/paimon/pull/8252#discussion_r3457410992
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/execution/PaimonStrategy.scala:
##########
@@ -215,3 +238,262 @@ case class PaimonStrategy(spark: SparkSession)
SparkShimLoader.shim.classicApi.recacheByPlan(spark, v2Relation)
}
}
+
+case class LateralVectorSearchExec(
+ innerTable: InnerTable,
+ columnName: String,
+ queryVectorExpr: Expression,
+ limit: Int,
+ options: Map[String, String],
+ vectorSearchOutput: Seq[Attribute],
+ projectList: Seq[NamedExpression],
+ projectOutput: Seq[Attribute],
+ child: SparkPlan)
+ extends SparkPlan {
+
+ override def children: Seq[SparkPlan] = Seq(child)
+
+ override def output: Seq[Attribute] = child.output ++ projectOutput
+
+ @transient override lazy val producedAttributes: AttributeSet = AttributeSet(vectorSearchOutput)
+
+ @transient
+ override lazy val references: AttributeSet = {
+ AttributeSet.fromAttributeSets(expressions.map(_.references)) -- producedAttributes
+ }
+
+ override protected def withNewChildrenInternal(newChildren: IndexedSeq[SparkPlan]): SparkPlan = {
+ copy(child = newChildren.head)
+ }
+
+ override protected def doExecute(): RDD[InternalRow] = {
+ child.execute().mapPartitions {
+ outerRows =>
+ val strippedQueryExpr = queryVectorExpr.transform {
+ case OuterReference(namedExpression) => namedExpression.toAttribute
+ }
+ val queryVectorProjection = UnsafeProjection.create(Seq(strippedQueryExpr), child.output)
+ val rightProjection = UnsafeProjection.create(projectList, vectorSearchOutput)
+ val joinedRow = new JoinedRow
+ lazy val searchContext = createSearchContext(rightProjection)
+ val batchSize = searchContext.batchSize
+
+ outerRows.map(_.copy()).grouped(batchSize).flatMap {
+ outerRowBatch =>
+ val searchBatch = ArrayBuffer[LateralVectorSearchQuery]()
+ outerRowBatch.foreach {
+ outerRow =>
+ toFloatArray(queryVectorProjection(outerRow).get(0, strippedQueryExpr.dataType))
+ .foreach(
+ queryVector => searchBatch += LateralVectorSearchQuery(outerRow, queryVector))
+ }
+
+ if (searchBatch.isEmpty) {
+ Iterator.empty
+ } else {
+ search(searchBatch.toVector, searchContext).map {
+ case (outerRow, rightRow) =>
+ joinedRow(outerRow, rightRow)
+ joinedRow.copy()
+ }
+ }
+ }
+ }
+ }
+
+ private def createSearchContext(rightProjection: UnsafeProjection): LateralVectorSearchContext = {
+ val rowType = innerTable.rowType()
+ val readFieldNames = vectorSearchOutput
+ .filterNot(_.name == PaimonMetadataColumn.SEARCH_SCORE_COLUMN)
+ .map(_.name)
+ val readFieldNamesWithRowId =
+ if (readFieldNames.contains(SpecialFields.ROW_ID.name())) {
+ readFieldNames
+ } else {
+ readFieldNames :+ SpecialFields.ROW_ID.name()
+ }
+ val rowTypeWithRowId = SpecialFields.rowTypeWithRowId(rowType)
+ val readRowType = rowType.project(readFieldNames.asJava)
+ val readRowTypeWithRowId = SpecialFields.rowTypeWithRowId(readRowType)
+ val readBuilder = innerTable
+ .newReadBuilder()
+ .withReadType(rowTypeWithRowId.project(readFieldNamesWithRowId.asJava))
+ val scoreMetadataColumns =
+ if (vectorSearchOutput.exists(_.name == PaimonMetadataColumn.SEARCH_SCORE_COLUMN)) {
+ Seq(PaimonMetadataColumn.SEARCH_SCORE)
+ } else {
+ Seq.empty
+ }
+ val resultRowType =
+ if (scoreMetadataColumns.isEmpty) {
+ readRowTypeWithRowId
+ } else {
+ new RowType(
+ (readRowTypeWithRowId.getFields.asScala ++ scoreMetadataColumns.map(
+ _.toPaimonDataField)).asJava)
+ }
+ val sparkRow = SparkInternalRow.create(resultRowType)
+ val vectorSearchBuilder = innerTable
+ .newBatchVectorSearchBuilder()
+ .withVectorColumn(columnName)
+ .withLimit(limit)
+ .withOptions(options.asJava)
+
+ val vectorPlan = vectorSearchBuilder.newVectorScan().scan()
+ val batchSize =
+ Math.max(1, new CoreOptions(innerTable.options()).vectorSearchLateralJoinBatchSize())
+
+ LateralVectorSearchContext(
+ readBuilder,
+ vectorSearchBuilder,
+ vectorPlan,
+ scoreMetadataColumns,
+ sparkRow,
+ rowIdOrdinal = resultRowType.getFieldIndex(SpecialFields.ROW_ID.name()),
+ projectionInputOrdinals = vectorSearchOutput.map {
+ attr =>
+ if (attr.name == PaimonMetadataColumn.SEARCH_SCORE_COLUMN) {
+ -1
+ } else {
+ resultRowType.getFieldIndex(attr.name)
+ }
+ },
+ rightProjection,
+ batchSize
+ )
+ }
+
+ private def search(
+ queries: Seq[LateralVectorSearchQuery],
+ context: LateralVectorSearchContext): Iterator[(InternalRow, InternalRow)] = {
+ val vectors = queries.map(_.queryVector).toArray
+ val globalIndexResults = context.vectorSearchBuilder
+ .withVectors(vectors)
+ .newBatchVectorRead()
+ .readBatch(context.vectorPlan)
+ .asScala
+ .toVector
+ val rowIdToMatches = createRowIdToMatches(queries, globalIndexResults)
+ val batchGlobalIndexResult = createBatchGlobalIndexResult(globalIndexResults)
+ val scan = context.readBuilder
+ .newScan()
+ .withGlobalIndexResult(batchGlobalIndexResult)
+ .asInstanceOf[InnerTableScan]
+ val read = context.readBuilder.newRead()
+
+ scan.plan().splits().asScala.iterator.flatMap {
+ split =>
+ val reader =
Review Comment:
This reader is only closed when this inner iterator is fully exhausted. If a downstream operator short-circuits consumption, for example `LIMIT 1`/`take`, or if the task is interrupted, Spark can stop pulling rows before `hasNext` returns false, leaving the current `PaimonRecordReaderIterator` and its underlying `RecordReader` open. Please register a `TaskContext` completion listener or wrap the returned iterator so the current reader is closed on task completion/cancellation as well as normal exhaustion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] support lateral inner join for vector search [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8252:
URL: https://github.com/apache/paimon/pull/8252#discussion_r3460434065
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/plans/logical/PaimonTableValuedFunctions.scala:
##########
@@ -189,6 +194,57 @@ object PaimonTableValuedFunctions {
}
}
+ def resolveLateralVectorSearch(
+ left: LogicalPlan,
+ right: LogicalPlan,
+ joinType: JoinType,
+ condition: Option[Expression]): Option[LogicalPlan] = {
+ extractDynamicVectorSearch(right) match {
+ case None =>
+ None
+ case Some(_) if joinType != Inner =>
+ throw new RuntimeException(
+ s"LATERAL vector_search only supports INNER join, but got: $joinType.")
+ case Some((relation, projectList, projectOutput)) =>
+ val vectorSearchOutput = vectorSearchOutputForProject(relation, projectList)
+ val lateralVectorSearch =
+ LateralVectorSearch(
+ left,
+ relation.innerTable,
+ relation.columnName,
+ relation.queryVectorExpr,
+ relation.limit,
+ relation.options,
+ vectorSearchOutput,
+ projectList,
+ projectOutput)
+ Some(condition.map(Filter(_, lateralVectorSearch)).getOrElse(lateralVectorSearch))
+ }
+ }
+
+ private def vectorSearchOutputForProject(
+ relation: DynamicVectorSearchRelation,
+ projectList: Seq[NamedExpression]): Seq[Attribute] = {
+ val projectReferences = AttributeSet.fromAttributeSets(projectList.map(_.references))
+ relation.output.filter(projectReferences.contains)
+ }
+
+ private def extractDynamicVectorSearch(plan: LogicalPlan)
Review Comment:
Filters inside the lateral subquery are still not handled here. For example, `FROM q, LATERAL (SELECT gid FROM vector_search('t', 'embs', q.embs, 5) WHERE dt = '20260608') r` is resolved as a right plan containing `Filter(..., DynamicVectorSearchRelation)` (usually under `Project`). This extractor falls through to `None`, leaving the dynamic relation with an outer reference but no `LateralVectorSearch` physical path. Please extract `Filter` nodes here and append their conditions to `searchFilters` (or reject them explicitly) in addition to the outer-`WHERE` pushdown case.
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/optimizer/PushDownLateralVectorSearchFilter.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.spark.catalyst.optimizer
+
+import org.apache.paimon.spark.catalyst.plans.logical.LateralVectorSearch
+
+import org.apache.spark.sql.catalyst.expressions.{And, PredicateHelper}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/** Pushes filters on the query side below lateral vector search. */
+object PushDownLateralVectorSearchFilter extends Rule[LogicalPlan] with PredicateHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.transform {
+ case filter @ Filter(condition, lvs: LateralVectorSearch) =>
+ val predicates = splitConjunctivePredicates(condition)
+ val (pushDownToLeft, otherPredicates) = predicates.partition {
+ predicate => predicate.deterministic && predicate.references.subsetOf(lvs.child.outputSet)
+ }
+ val (pushDownToSearch, stayUp) = otherPredicates.partition {
Review Comment:
This removes every deterministic predicate that references only search-side output, but `convertSearchFilters()` later throws if Spark cannot translate the rewritten expression into a Paimon predicate. A valid query such as `WHERE r.gid + 1 > 10`, or a filter on an expression alias from the lateral subquery, would now fail during execution instead of being evaluated above the lateral result. Please keep untranslatable predicates in `stayUp`, or restrict this pushdown to simple field predicates that are known to be convertible before dropping the upper filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] support lateral inner join for vector search [paimon]
Posted by "Stefanietry (via GitHub)" <gi...@apache.org>.
Stefanietry closed pull request #8252: [spark] support lateral inner join for vector search
URL: https://github.com/apache/paimon/pull/8252
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] support lateral inner join for vector search [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8252:
URL: https://github.com/apache/paimon/pull/8252#discussion_r3463830606
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/plans/logical/PaimonTableValuedFunctions.scala:
##########
@@ -447,6 +592,48 @@ case class VectorSearchQuery(override val args: Seq[Expression])
}
value.toString
}
+
+ def hasOuterReference(argsWithoutTable: Seq[Expression]): Boolean = {
+ val queryVector = argsWithoutTable(1)
Review Comment:
hasOuterReference accesses argsWithoutTable(1) before verifying the arity. For invalid calls such as vector_search('t', 'embs'), this now throws IndexOutOfBoundsException during resolution instead of the existing helpful `vector_search needs three or four parameters...` error from createVectorSearch/createDynamicVectorSearch. Please check the size before reading the query-vector argument.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] support lateral inner join for vector search [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8252:
URL: https://github.com/apache/paimon/pull/8252#issuecomment-4720304546
Please fix test failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] support lateral inner join for vector search [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8252:
URL: https://github.com/apache/paimon/pull/8252#discussion_r3457409791
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/execution/PaimonStrategy.scala:
##########
@@ -215,3 +238,262 @@ case class PaimonStrategy(spark: SparkSession)
SparkShimLoader.shim.classicApi.recacheByPlan(spark, v2Relation)
}
}
+
+case class LateralVectorSearchExec(
+ innerTable: InnerTable,
+ columnName: String,
+ queryVectorExpr: Expression,
+ limit: Int,
+ options: Map[String, String],
+ vectorSearchOutput: Seq[Attribute],
+ projectList: Seq[NamedExpression],
+ projectOutput: Seq[Attribute],
+ child: SparkPlan)
+ extends SparkPlan {
+
+ override def children: Seq[SparkPlan] = Seq(child)
+
+ override def output: Seq[Attribute] = child.output ++ projectOutput
+
+ @transient override lazy val producedAttributes: AttributeSet = AttributeSet(vectorSearchOutput)
+
+ @transient
+ override lazy val references: AttributeSet = {
+ AttributeSet.fromAttributeSets(expressions.map(_.references)) -- producedAttributes
+ }
+
+ override protected def withNewChildrenInternal(newChildren: IndexedSeq[SparkPlan]): SparkPlan = {
+ copy(child = newChildren.head)
+ }
+
+ override protected def doExecute(): RDD[InternalRow] = {
+ child.execute().mapPartitions {
+ outerRows =>
+ val strippedQueryExpr = queryVectorExpr.transform {
+ case OuterReference(namedExpression) => namedExpression.toAttribute
+ }
+ val queryVectorProjection = UnsafeProjection.create(Seq(strippedQueryExpr), child.output)
+ val rightProjection = UnsafeProjection.create(projectList, vectorSearchOutput)
+ val joinedRow = new JoinedRow
+ lazy val searchContext = createSearchContext(rightProjection)
+ val batchSize = searchContext.batchSize
+
+ outerRows.map(_.copy()).grouped(batchSize).flatMap {
+ outerRowBatch =>
+ val searchBatch = ArrayBuffer[LateralVectorSearchQuery]()
+ outerRowBatch.foreach {
+ outerRow =>
+ toFloatArray(queryVectorProjection(outerRow).get(0, strippedQueryExpr.dataType))
+ .foreach(
+ queryVector => searchBatch += LateralVectorSearchQuery(outerRow, queryVector))
+ }
+
+ if (searchBatch.isEmpty) {
+ Iterator.empty
+ } else {
+ search(searchBatch.toVector, searchContext).map {
+ case (outerRow, rightRow) =>
+ joinedRow(outerRow, rightRow)
+ joinedRow.copy()
+ }
+ }
+ }
+ }
+ }
+
+ private def createSearchContext(rightProjection: UnsafeProjection): LateralVectorSearchContext = {
+ val rowType = innerTable.rowType()
+ val readFieldNames = vectorSearchOutput
+ .filterNot(_.name == PaimonMetadataColumn.SEARCH_SCORE_COLUMN)
+ .map(_.name)
+ val readFieldNamesWithRowId =
+ if (readFieldNames.contains(SpecialFields.ROW_ID.name())) {
+ readFieldNames
+ } else {
+ readFieldNames :+ SpecialFields.ROW_ID.name()
+ }
+ val rowTypeWithRowId = SpecialFields.rowTypeWithRowId(rowType)
+ val readRowType = rowType.project(readFieldNames.asJava)
+ val readRowTypeWithRowId = SpecialFields.rowTypeWithRowId(readRowType)
+ val readBuilder = innerTable
+ .newReadBuilder()
+ .withReadType(rowTypeWithRowId.project(readFieldNamesWithRowId.asJava))
+ val scoreMetadataColumns =
+ if (vectorSearchOutput.exists(_.name == PaimonMetadataColumn.SEARCH_SCORE_COLUMN)) {
+ Seq(PaimonMetadataColumn.SEARCH_SCORE)
+ } else {
+ Seq.empty
+ }
+ val resultRowType =
+ if (scoreMetadataColumns.isEmpty) {
+ readRowTypeWithRowId
+ } else {
+ new RowType(
+ (readRowTypeWithRowId.getFields.asScala ++ scoreMetadataColumns.map(
+ _.toPaimonDataField)).asJava)
+ }
+ val sparkRow = SparkInternalRow.create(resultRowType)
+ val vectorSearchBuilder = innerTable
Review Comment:
Normal Spark `vector_search` scans apply pushed partition/data filters before top-k (`PaimonBaseScan.evalVectorSearch` passes `pushedPartitionFilters`/`pushedDataFilters` into the builder). This lateral executor builds the `BatchVectorSearchBuilder` here without carrying predicates from the search side; `PushDownLateralVectorSearchFilter` only pushes predicates that reference the left child, so predicates on `r.dt` or other searched-table columns stay above `LateralVectorSearch`. A query like `... JOIN LATERAL (...) r WHERE r.dt = '20260420'` will pick topK over all partitions and then filter the joined rows, which can return fewer or wrong rows compared with non-lateral `vector_search(...) WHERE dt = ...`. Please preserve search-side filters and apply them via `withPartitionFilter`/`withFilter` before `newVectorScan()`/`readBatch()`, or reject such predicates explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] support lateral inner join for vector search [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8252:
URL: https://github.com/apache/paimon/pull/8252
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org