You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@auron.apache.org by "weimingdiit (via GitHub)" <gi...@apache.org> on 2026/04/04 10:02:05 UTC
[PR] [AURON #2163] Support native Iceberg scans with residual filters via scan pruning and post-scan native filter [auron]
weimingdiit opened a new pull request, #2164:
URL: https://github.com/apache/auron/pull/2164
# Which issue does this PR close?
Closes #2163
# Rationale for this change
The previous behavior was too conservative for Iceberg scans with residual filters. Even when the scan could still be executed natively and the remaining filter logic could be handled above the scan, the planner would fall back entirely.
This PR improves native coverage for Iceberg reads by:
- preserving correctness for unsupported predicates
- increasing native scan applicability for common filter patterns
- reusing the existing native filter path instead of requiring full scan-level predicate support up front
This is an incremental improvement to Iceberg native execution, not full Iceberg feature coverage.
# What changes are included in this PR?
This PR:
- removes the unconditional fallback for Iceberg scans with non-`alwaysTrue` residual filters
- extends `IcebergScanPlan` to carry `pruningPredicates`
- extracts Iceberg scan filter expressions and converts a supported subset into Spark expressions
- converts those Spark expressions into native scan pruning predicates
- passes pruning predicates down through `NativeIcebergTableScanExec`
- keeps unsupported predicates on the upper `NativeFilter` path
- adds integration coverage for:
- equality-based pruning
- `IN`-based pruning
- partial pushdown where only part of the predicate is pushed to scan pruning
- includes a small UI/build cleanup in `AuronAllExecutionsPage.scala` to avoid unrelated compile issues in the affected build/test path
## Supported predicate scope in this PR
The scan-pruning conversion added here supports a limited subset of Iceberg expressions, including:
- `AND`
- `OR`
- `NOT`
- `IS NULL`
- `IS NOT NULL`
- `IS NAN`
- `NOT NAN`
- comparison predicates such as `=`, `!=`, `<`, `<=`, `>`, `>=`
- `IN`
- `NOT IN`
The current implementation intentionally avoids pushing some types through scan pruning, including:
- `StringType`
- `BinaryType`
- `DecimalType`
Unsupported predicates are not pushed into scan pruning and are instead left for post-scan native filtering.
# How was this patch tested?
Integration coverage was added in `AuronIcebergIntegrationSuite`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2163] Support native Iceberg scans with residual filters via scan pruning and post-scan native filter [auron]
Posted by "weimingdiit (via GitHub)" <gi...@apache.org>.
weimingdiit commented on PR #2164:
URL: https://github.com/apache/auron/pull/2164#issuecomment-4227758438
@slfan1989 @yew1eb Thanks for your review and merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2163] Support native Iceberg scans with residual filters via scan pruning and post-scan native filter [auron]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2164:
URL: https://github.com/apache/auron/pull/2164#discussion_r3036344985
##########
thirdparty/auron-iceberg/src/test/scala/org/apache/auron/iceberg/AuronIcebergIntegrationSuite.scala:
##########
@@ -132,14 +134,77 @@ class AuronIcebergIntegrationSuite
}
}
- test("iceberg scan falls back for residual filters on data columns") {
+ test("iceberg scan pushes residual filters into native scan pruning predicates") {
withTable("local.db.t_residual") {
sql("create table local.db.t_residual (id int, v string) using iceberg")
sql("insert into local.db.t_residual values (1, 'a'), (2, 'b')")
- val df = sql("select * from local.db.t_residual where v = 'a'")
+ val df = sql("select * from local.db.t_residual where id = 1")
checkAnswer(df, Seq(Row(1, "a")))
+ val nativeScanPlan = icebergScanPlan(df)
+ assert(nativeScanPlan.nonEmpty)
+ assert(nativeScanPlan.get.pruningPredicates.nonEmpty)
val plan = df.queryExecution.executedPlan.toString()
- assert(!plan.contains("NativeIcebergTableScan"))
+ assert(plan.contains("NativeIcebergTableScan"))
+ assert(plan.contains("NativeFilter"))
+ }
+ }
+
+ test("iceberg scan pushes supported IN filters into native scan pruning predicates") {
+ withTable("local.db.t_residual_supported") {
+ sql("create table local.db.t_residual_supported (id int, v string) using iceberg")
+ sql(
+ "insert into local.db.t_residual_supported values (1, 'alpha'), (2, 'beta'), (3, 'atom')")
+ val df = sql("""
+ |select * from local.db.t_residual_supported
+ |where id in (1, 3)
+ |""".stripMargin)
+ checkAnswer(df, Seq(Row(1, "alpha"), Row(3, "atom")))
+ val nativeScanPlan = icebergScanPlan(df)
+ assert(nativeScanPlan.nonEmpty)
+ assert(nativeScanPlan.get.pruningPredicates.nonEmpty)
+ val plan = df.queryExecution.executedPlan.toString()
+ assert(plan.contains("NativeIcebergTableScan"))
+ assert(plan.contains("NativeFilter"))
+ }
+ }
+
+ test("iceberg scan keeps native post-scan filter when only part of the predicate is pushed") {
+ withTable("local.db.t_residual_partial_pushdown") {
+ sql("create table local.db.t_residual_partial_pushdown (id int, v string) using iceberg")
+ sql(
+ "insert into local.db.t_residual_partial_pushdown values (1, 'alpha'), (2, 'beta'), (3, 'atom'), (4, 'delta')")
+ val df = sql("""
+ |select * from local.db.t_residual_partial_pushdown
+ |where id in (1, 2, 3) and id % 2 = 1
+ |""".stripMargin)
+ checkAnswer(df, Seq(Row(1, "alpha"), Row(3, "atom")))
+ val nativeScanPlan = icebergScanPlan(df)
+ assert(nativeScanPlan.nonEmpty)
+ assert(nativeScanPlan.get.pruningPredicates.nonEmpty)
+ val plan = df.queryExecution.executedPlan.toString()
+ assert(plan.contains("NativeIcebergTableScan"))
+ assert(plan.contains("NativeFilter"))
+ }
+ }
+
+ test("iceberg scan keeps native string filter outside scan pruning") {
+ withTable("local.db.t_residual_string") {
+ sql("create table local.db.t_residual_string (id int, v string) using iceberg")
+ sql("insert into local.db.t_residual_string values (1, 'a'), (2, 'b'), (3, null)")
+ val df = sql("""
+ |select * from local.db.t_residual_string
+ |where v = 'a'
+ |""".stripMargin)
+ checkAnswer(df, Seq(Row(1, "a")))
+ val nativeScanPlan = icebergScanPlan(df)
+ assert(nativeScanPlan.nonEmpty)
+ assert(nativeScanPlan.get.pruningPredicates.nonEmpty)
+ assert(
+ !nativeScanPlan.get.pruningPredicates.exists(_.toString.contains("binary_expr")),
+ "string equality should remain on the post-scan native filter path")
+ val plan = df.queryExecution.executedPlan.toString()
+ assert(plan.contains("NativeIcebergTableScan"))
+ assert(plan.contains("NativeFilter"))
Review Comment:
This test inspects `pruningPredicates` by calling `toString` on protobuf nodes and searching for the substring `"binary_expr"`. Protobuf `toString` output is not a stable API (it can change with proto/ScalaPB versions), making the assertion brittle. Prefer asserting on the actual protobuf oneof/case fields (e.g., the expr-node kind) or on higher-level properties (like which Spark expressions were converted) rather than string-matching the debug representation.
```suggestion
val plan = df.queryExecution.executedPlan.toString()
assert(plan.contains("NativeIcebergTableScan"))
assert(plan.contains("NativeFilter"),
"string equality should remain on the post-scan native filter path")
```
##########
thirdparty/auron-iceberg/src/main/scala/org/apache/spark/sql/auron/iceberg/IcebergScanSupport.scala:
##########
@@ -188,4 +190,240 @@ object IcebergScanSupport extends Logging {
None
}
}
+
+ private def collectPruningPredicates(
+ scan: AnyRef,
+ readSchema: StructType): Seq[pb.PhysicalExprNode] = {
+ scanFilterExpressions(scan).flatMap { expr =>
+ convertIcebergFilterExpression(expr, readSchema) match {
+ case Some(converted) =>
+ Some(NativeConverters.convertScanPruningExpr(converted))
+ case None =>
+ logDebug(s"Skip unsupported Iceberg pruning expression: $expr")
+ None
+ }
+ }
+ }
+
+ private def scanFilterExpressions(scan: AnyRef): Seq[IcebergExpression] = {
+ invokeDeclaredMethod(scan, "filterExpressions") match {
+ case Some(values: java.util.Collection[_]) =>
+ values.asScala.collect { case expr: IcebergExpression => expr }.toSeq
+ case Some(values: Seq[_]) =>
+ values.collect { case expr: IcebergExpression => expr }
+ case _ =>
+ Seq.empty
+ }
+ }
+
+ private def invokeDeclaredMethod(target: AnyRef, methodName: String): Option[Any] = {
+ try {
+ var cls: Class[_] = target.getClass
+ while (cls != null) {
+ cls.getDeclaredMethods.find(_.getName == methodName) match {
+ case Some(method) =>
+ method.setAccessible(true)
+ return Some(method.invoke(target))
+ case None =>
Review Comment:
`invokeDeclaredMethod` picks the first declared method whose name matches, without verifying it is a zero-arg method. If the target class has an overloaded `filterExpressions` (or a synthetic bridge method) that takes parameters, `method.invoke(target)` will throw and silently disable pruning. Consider selecting only methods with `getParameterCount == 0` (and ideally `getReturnType` compatible with a collection) before invoking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2163] Support native Iceberg scans with residual filters via scan pruning and post-scan native filter [auron]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #2164:
URL: https://github.com/apache/auron/pull/2164#issuecomment-4227363722
@weimingdiit Thanks for the contribution! @yew1eb Thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2163] Support native Iceberg scans with residual filters via scan pruning and post-scan native filter [auron]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 merged PR #2164:
URL: https://github.com/apache/auron/pull/2164
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org