You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "fresh-borzoni (via GitHub)" <gi...@apache.org> on 2026/05/02 03:16:55 UTC
[PR] [spark] lake table predicate pushdown [fluss]
fresh-borzoni opened a new pull request, #3238:
URL: https://github.com/apache/fluss/pull/3238
## Contribution Checklist
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] lake table predicate pushdown [fluss]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on PR #3238:
URL: https://github.com/apache/fluss/pull/3238#issuecomment-4397429505
@Yohahaha Ty for the review, addressed
@luoyuxia Can you take a quick look, pls?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] lake table predicate pushdown [fluss]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni merged PR #3238:
URL: https://github.com/apache/fluss/pull/3238
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] lake table predicate pushdown [fluss]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on code in PR #3238:
URL: https://github.com/apache/fluss/pull/3238#discussion_r3201654411
##########
fluss-spark/fluss-spark-common/src/main/scala/org/apache/fluss/spark/read/lake/FlussLakeBatch.scala:
##########
@@ -62,3 +67,17 @@ abstract class FlussLakeBatch(
.toMap
}
}
+
+object FlussLakeBatch {
+
+ private val LOG = LoggerFactory.getLogger(classOf[FlussLakeBatch])
+
+ def applyLakeFilters(lakeSource: LakeSource[LakeSplit], predicate: FlussPredicate): Unit = {
+ val result = lakeSource.withFilters(Collections.singletonList(predicate))
+ if (LOG.isDebugEnabled) {
+ val accepted = result.acceptedPredicates().size()
+ val remaining = result.remainingPredicates().size()
Review Comment:
Good point 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] lake table predicate pushdown [fluss]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on code in PR #3238:
URL: https://github.com/apache/fluss/pull/3238#discussion_r3208712570
##########
fluss-spark/fluss-spark-common/src/main/scala/org/apache/fluss/spark/read/lake/FlussLakeBatch.scala:
##########
@@ -74,11 +74,9 @@ object FlussLakeBatch extends Logging {
lakeSource: LakeSource[LakeSplit],
predicates: java.util.List[FlussPredicate]): LakeSource.FilterPushDownResult = {
val result = lakeSource.withFilters(predicates)
- logDebug {
- val accepted = result.acceptedPredicates().size()
- val remaining = result.remainingPredicates().size()
- s"Lake source accepted $accepted of ${accepted + remaining} pushed predicates"
- }
+ logDebug(
Review Comment:
nit: it can be info level
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] lake table predicate pushdown [fluss]
Posted by "Yohahaha (via GitHub)" <gi...@apache.org>.
Yohahaha commented on code in PR #3238:
URL: https://github.com/apache/fluss/pull/3238#discussion_r3192801049
##########
fluss-spark/fluss-spark-common/src/main/scala/org/apache/fluss/spark/read/lake/FlussLakeBatch.scala:
##########
@@ -62,3 +67,17 @@ abstract class FlussLakeBatch(
.toMap
}
}
+
+object FlussLakeBatch {
+
+ private val LOG = LoggerFactory.getLogger(classOf[FlussLakeBatch])
+
+ def applyLakeFilters(lakeSource: LakeSource[LakeSplit], predicate: FlussPredicate): Unit = {
+ val result = lakeSource.withFilters(Collections.singletonList(predicate))
+ if (LOG.isDebugEnabled) {
+ val accepted = result.acceptedPredicates().size()
+ val remaining = result.remainingPredicates().size()
Review Comment:
could we use remaining filters update pushedPredicate?
##########
fluss-spark/fluss-spark-common/src/main/scala/org/apache/fluss/spark/read/lake/FlussLakeBatch.scala:
##########
@@ -62,3 +67,17 @@ abstract class FlussLakeBatch(
.toMap
}
}
+
+object FlussLakeBatch {
+
+ private val LOG = LoggerFactory.getLogger(classOf[FlussLakeBatch])
Review Comment:
use spark Logging interface
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] lake table predicate pushdown [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #3238:
URL: https://github.com/apache/fluss/pull/3238#discussion_r3208317091
##########
fluss-spark/fluss-spark-common/src/main/scala/org/apache/fluss/spark/read/lake/FlussLakeBatch.scala:
##########
@@ -62,3 +67,23 @@ abstract class FlussLakeBatch(
.toMap
}
}
+
+object FlussLakeBatch extends Logging {
+
+ def applyLakeFilters(
+ lakeSource: LakeSource[LakeSplit],
+ predicates: java.util.List[FlussPredicate]): LakeSource.FilterPushDownResult = {
+ val result = lakeSource.withFilters(predicates)
+ logDebug {
+ val accepted = result.acceptedPredicates().size()
+ val remaining = result.remainingPredicates().size()
+ s"Lake source accepted $accepted of ${accepted + remaining} pushed predicates"
Review Comment:
nit: whether it'll be better to also print what predicates are accepected, and what not for better track?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] lake table predicate pushdown [fluss]
Posted by "fresh-borzoni (via GitHub)" <gi...@apache.org>.
fresh-borzoni commented on PR #3238:
URL: https://github.com/apache/fluss/pull/3238#issuecomment-4363605997
@Yohahaha @YannByron PTAL 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] lake table predicate pushdown [fluss]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3238:
URL: https://github.com/apache/fluss/pull/3238#discussion_r3206874101
##########
fluss-spark/fluss-spark-ut/src/test/scala/org/apache/fluss/spark/lake/SparkLakeLogTableReadTest.scala:
##########
@@ -398,6 +398,132 @@ abstract class SparkLakeLogTableReadTest extends SparkLakeTableReadTestBase {
}
}
+ test("Spark Lake Read: filter pushdown — lake-only (all data in lake)") {
+ withTable("t_pd_lake") {
+ sql(s"""
+ |CREATE TABLE $DEFAULT_DATABASE.t_pd_lake (id INT, amount INT, name STRING)
+ | TBLPROPERTIES (
+ | '${ConfigOptions.TABLE_DATALAKE_ENABLED.key()}' = true,
+ | '${ConfigOptions.TABLE_DATALAKE_FRESHNESS.key()}' = '1s',
+ | '${BUCKET_NUMBER.key()}' = 1)
+ |""".stripMargin)
+ sql(s"""
+ |INSERT INTO $DEFAULT_DATABASE.t_pd_lake VALUES
+ |(1, 100, 'alpha'), (2, 200, 'beta'), (3, 300, 'gamma')
+ |""".stripMargin)
+ tierToLake("t_pd_lake")
+
+ val query =
+ sql(s"SELECT * FROM $DEFAULT_DATABASE.t_pd_lake WHERE amount > 150 ORDER BY id")
+ checkAnswer(query, Row(2, 200, "beta") :: Row(3, 300, "gamma") :: Nil)
+ assertPushedNames(query, Set(">"))
+ }
+ }
+
+ test("Spark Lake Read: filter pushdown — union (lake + log tail)") {
+ withTable("t_pd_union") {
+ sql(s"""
+ |CREATE TABLE $DEFAULT_DATABASE.t_pd_union (id INT, amount INT, name STRING)
+ | TBLPROPERTIES (
+ | '${ConfigOptions.TABLE_DATALAKE_ENABLED.key()}' = true,
+ | '${ConfigOptions.TABLE_DATALAKE_FRESHNESS.key()}' = '1s',
+ | '${BUCKET_NUMBER.key()}' = 1)
+ |""".stripMargin)
+ sql(s"""
+ |INSERT INTO $DEFAULT_DATABASE.t_pd_union VALUES
+ |(1, 100, 'alpha'), (2, 200, 'beta'), (3, 300, 'gamma')
+ |""".stripMargin)
+ tierToLake("t_pd_union")
+ sql(s"""
+ |INSERT INTO $DEFAULT_DATABASE.t_pd_union VALUES
+ |(4, 400, 'delta'), (5, 500, 'epsilon')
+ |""".stripMargin)
+
+ val query =
+ sql(s"SELECT * FROM $DEFAULT_DATABASE.t_pd_union WHERE amount >= 200 ORDER BY id")
+ checkAnswer(
+ query,
+ Row(2, 200, "beta") ::
+ Row(3, 300, "gamma") ::
+ Row(4, 400, "delta") ::
+ Row(5, 500, "epsilon") :: Nil
+ )
+ assertPushedNames(query, Set(">="))
+ }
+ }
+
+ test("Spark Lake Read: filter pushdown — fallback (no lake snapshot)") {
+ withTable("t_pd_fallback") {
+ sql(s"""
+ |CREATE TABLE $DEFAULT_DATABASE.t_pd_fallback (id INT, amount INT, name STRING)
+ | TBLPROPERTIES (
+ | '${ConfigOptions.TABLE_DATALAKE_ENABLED.key()}' = true,
+ | '${ConfigOptions.TABLE_DATALAKE_FRESHNESS.key()}' = '1s',
+ | '${BUCKET_NUMBER.key()}' = 1)
+ |""".stripMargin)
+ sql(s"""
+ |INSERT INTO $DEFAULT_DATABASE.t_pd_fallback VALUES
+ |(1, 100, 'alpha'), (2, 200, 'beta'), (3, 300, 'gamma')
+ |""".stripMargin)
+
+ val query =
+ sql(s"SELECT * FROM $DEFAULT_DATABASE.t_pd_fallback WHERE amount = 200")
+ checkAnswer(query, Row(2, 200, "beta") :: Nil)
+ assertPushedNames(query, Set("="))
+ }
+ }
+
+ test("Spark Lake Read: filter pushdown — partitioned lake table") {
+ withTable("t_pd_partitioned") {
+ sql(s"""
+ |CREATE TABLE $DEFAULT_DATABASE.t_pd_partitioned
+ | (id INT, amount INT, name STRING, dt STRING)
+ | PARTITIONED BY (dt)
+ | TBLPROPERTIES (
+ | '${ConfigOptions.TABLE_DATALAKE_ENABLED.key()}' = true,
+ | '${ConfigOptions.TABLE_DATALAKE_FRESHNESS.key()}' = '1s',
+ | '${BUCKET_NUMBER.key()}' = 1)
+ |""".stripMargin)
+ sql(s"""
+ |INSERT INTO $DEFAULT_DATABASE.t_pd_partitioned VALUES
+ |(1, 100, 'alpha', '2026-01-01'),
+ |(2, 200, 'beta', '2026-01-01'),
+ |(3, 300, 'gamma', '2026-01-02')
+ |""".stripMargin)
+ tierToLake("t_pd_partitioned")
+
+ val query = sql(s"""
+ |SELECT id, amount, dt FROM $DEFAULT_DATABASE.t_pd_partitioned
+ |WHERE amount >= 200 ORDER BY id""".stripMargin)
+ checkAnswer(query, Row(2, 200, "2026-01-01") :: Row(3, 300, "2026-01-02") :: Nil)
+ assertPushedNames(query, Set(">="))
+ }
+ }
+
+ test("Spark Lake Read: filter pushdown — non-pushable predicate falls back to Spark") {
+ withTable("t_pd_nonpushable") {
+ sql(s"""
+ |CREATE TABLE $DEFAULT_DATABASE.t_pd_nonpushable (id INT, amount INT, name STRING)
+ | TBLPROPERTIES (
+ | '${ConfigOptions.TABLE_DATALAKE_ENABLED.key()}' = true,
+ | '${ConfigOptions.TABLE_DATALAKE_FRESHNESS.key()}' = '1s',
+ | '${BUCKET_NUMBER.key()}' = 1)
+ |""".stripMargin)
+ sql(s"""
+ |INSERT INTO $DEFAULT_DATABASE.t_pd_nonpushable VALUES
+ |(1, 100, 'alpha'), (2, 200, 'beta'), (3, 300, 'gamma'), (4, 400, 'delta')
+ |""".stripMargin)
+ tierToLake("t_pd_nonpushable")
+
+ val query =
+ sql(s"SELECT id FROM $DEFAULT_DATABASE.t_pd_nonpushable WHERE amount % 200 = 0 ORDER BY id")
+ checkAnswer(query, Row(2) :: Row(4) :: Nil)
+ // The modulo expression is not convertible; nothing should be pushed.
+ val pushed = pushedPredicates(query).map(_.name())
+ assert(pushed.forall(_ != "%"))
Review Comment:
The test comment says “nothing should be pushed”, but the assertion only checks that no pushed predicate is named "%". This would still pass if some other predicate were incorrectly reported as pushed. To match the intent, assert that `pushedPredicates(query)` is empty (or equivalently that `pushed` is empty).
##########
fluss-spark/fluss-spark-common/src/main/scala/org/apache/fluss/spark/read/FlussScanBuilder.scala:
##########
@@ -45,21 +50,54 @@ trait FlussSupportsPushDownV2Filters extends FlussScanBuilder with SupportsPushD
protected var pushedPredicate: Option[FlussPredicate] = None
protected var acceptedPredicates: Array[Predicate] = Array.empty[Predicate]
+ protected def convertAndStorePredicates(predicates: Array[Predicate]): Unit = {
+ val (predicate, accepted) =
+ SparkPredicateConverter.convertPredicates(tableInfo.getRowType, predicates.toSeq)
+ pushedPredicate = predicate
+ acceptedPredicates = accepted.toArray
+ }
+
override def pushPredicates(predicates: Array[Predicate]): Array[Predicate] = {
// Server-side batch filter only supports ARROW; other log formats reject it.
if (tableInfo.getTableConfig.getLogFormat == LogFormat.ARROW) {
- val (predicate, accepted) =
- SparkPredicateConverter.convertPredicates(tableInfo.getRowType, predicates.toSeq)
- pushedPredicate = predicate
- acceptedPredicates = accepted.toArray
+ convertAndStorePredicates(predicates)
}
- // Server-side filter is batch-level only; Spark must re-apply for row-exact results.
predicates
}
override def pushedPredicates(): Array[Predicate] = acceptedPredicates
}
+/**
+ * Lake reads push to the lake source regardless of log format. Each convertible predicate is
+ * offered to the lake source individually; only the lake-accepted subset is reported back to Spark
+ * and combined into the predicate handed to the scan.
+ */
+trait FlussLakeSupportsPushDownV2Filters extends FlussSupportsPushDownV2Filters {
+
+ def tablePath: TablePath
+
+ override def pushPredicates(predicates: Array[Predicate]): Array[Predicate] = {
+ val pairs =
+ SparkPredicateConverter.convertPerPredicate(tableInfo.getRowType, predicates.toSeq)
+ if (pairs.nonEmpty) {
+ val lakeSource =
+ FlussLakeUtils.createLakeSource(tableInfo.getProperties.toMap, tablePath)
+ val result = FlussLakeBatch.applyLakeFilters(lakeSource, pairs.map(_._2).asJava)
+ // Identity-match: lake sources are expected to return the same instances they received.
+ val acceptedSet: JSet[FlussPredicate] =
+ Collections.newSetFromMap(new IdentityHashMap())
+ acceptedSet.addAll(result.acceptedPredicates())
+ val (acceptedSpark, acceptedFluss) = pairs.collect {
+ case (sp, fp) if acceptedSet.contains(fp) => (sp, fp)
+ }.unzip
+ pushedPredicate = SparkPredicateConverter.combineAnd(acceptedFluss)
+ acceptedPredicates = acceptedSpark.toArray
+ }
Review Comment:
`pushPredicates` only updates `pushedPredicate`/`acceptedPredicates` when `pairs.nonEmpty`. If Spark calls `pushPredicates` more than once (e.g., during re-planning) with an empty or different predicate set, the scan builder can retain stale pushdown state from the previous call. Consider resetting `pushedPredicate` to `None` and `acceptedPredicates` to `Array.empty` at the start of the method (or in the `pairs.isEmpty` branch) before returning residual predicates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org