You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "sunchao (via GitHub)" <gi...@apache.org> on 2026/09/19 04:25:14 UTC

[PR] fix: defer throwing literal casts to Spark at runtime [datafusion-comet]

sunchao opened a new pull request, #6035:
URL: https://github.com/apache/datafusion-comet/pull/6035

   ## Which issue does this PR close?
   
   No existing issue. This fixes the literal-cast planning failure described below.
   
   ## Rationale for this change
   
   Spark can retain a throwing ANSI cast of a literal inside a conditional branch that the query never visits. Comet evaluates literal casts while serializing the plan, so it can raise before Spark requests that branch.
   
   For example, with Parquet rows `(0, "0")` and `(1, "1")`, this query should return `0`:
   
   ```sql
   SELECT CAST(IF(id = 1, 'bad', value) AS INT)
   FROM cast_branch_rows
   LIMIT 1
   ```
   
   Spark pushes the cast into the conditional branches and retains the invalid literal cast for conditional evaluation. Eagerly evaluating it during Comet planning changes the successful query into an error.
   
   ## What changes are included in this PR?
   
   When evaluation of a literal cast throws a nonfatal exception, return a fallback with an explanation so Spark evaluates the expression when demanded. Successful literal casts keep the existing folding path.
   
   The regression checks the unvisited branch returns the first row while retaining a native Parquet scan. A second query demands the invalid branch and verifies both executions raise `CAST_INVALID_INPUT` with matching exception type and SQLSTATE.
   
   ## How are these changes tested?
   
   - Built the native library from current upstream sources with `make core`.
   - Full Spark 4.1 test compilation, Spotless, and Scalastyle pass.
   - The new regression fails on the unchanged upstream implementation with `CAST_INVALID_INPUT` during planning.
   - With the fix, the complete `CometScalarFunctionSuite` passes: 15 tests, no failures or aborted suites (`mvn test -Pspark-4.1 -Dtest=none -Dsuites=org.apache.comet.serde.CometScalarFunctionSuite`).
   - The `run-spark-4.1-tests` label requests the broader Spark SQL checks in CI.
   
   Codex assisted with adapting the existing fix and regression, source review, and validation.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: defer throwing literal casts to Spark at runtime [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #6035:
URL: https://github.com/apache/datafusion-comet/pull/6035#discussion_r4052348832


##########
spark/src/test/scala/org/apache/comet/serde/CometScalarFunctionSuite.scala:
##########
@@ -216,6 +219,41 @@ class CometScalarFunctionSuite extends CometTestBase {
     assertRejectReason(withContext, "CometScalarFunction", "evalContext")
   }
 
+  test("literal cast failure in an unvisited conditional branch does not fail planning") {

Review Comment:
   This suite is entirely `CometScalarFunction` ANSI-sensitivity unit tests — all 15 call `.convert(...)` directly, with no SparkSession, no Parquet and no query execution. This test is end-to-end and exercises `CometCast`, so `CometNativeCastSuite` looks like the natural home; it is already the canonical place for cast behavior and already imports `CometCast` constants for exactly this kind of assertion.



##########
spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:
##########
@@ -109,7 +111,15 @@ object CometCast
     val cometEvalMode = evalMode(cast)
     cast.child match {
       case _: Literal =>
-        exprToProtoInternal(Literal.create(cast.eval(), cast.dataType), inputs, binding)
+        val value =
+          try {
+            cast.eval()
+          } catch {
+            case NonFatal(_) =>
+              withFallbackReason(cast, "Literal cast requires Spark's conditional evaluation")
+              return None
+          }

Review Comment:
   This is the block that decides the expression falls all the way back to Spark. Because `getSupportLevel` returned `Compatible`, `exprToProtoInternal` took the branch that calls `convert` directly, so `dispatchIfFallback` is never reached and the whole projection leaves the Comet pipeline.
   
   Consider moving the decision into `getSupportLevel`: try the evaluation there, and on `NonFatal` return `Unsupported(Some(literalCastConditionalEvalReason))`. The framework then tries the JVM codegen dispatcher first — which can run `Cast.doGenCode` in-pipeline and raise only if the branch is actually visited — and falls back to Spark only if the dispatcher declines. `convert` can then keep its current unconditional `cast.eval()`, since the failing case no longer reaches it.
   
   Two smaller things on this same block:
   
   **The reason string should be a shared constant.** This file already establishes the convention, with the rationale spelled out at the top:
   
   ```scala
   // Shared with CometNativeCastSuite so the asserted reason cannot drift from production.
   private[comet] val negativeScaleDecimalToStringReason: String = ...
   ```
   
   Here the string is written literally in both production and the test. Since `checkSparkAnswerAndFallbackReasons` matches with `contains`, a future edit would not break compilation and would not necessarily fail the test either — exactly the drift that comment guards against. On the wording: "Literal cast requires Spark's conditional evaluation" states the remedy but not the cause, and a user seeing this in EXPLAIN is asking why their query left Comet. Something self-explanatory would read better, e.g. "Cast of a literal threw during planning; Spark leaves it for conditional evaluation so it may never be reached at runtime".
   
   **Please add a comment explaining why the catch exists.** The reason is genuinely non-obvious: Spark's `ConstantFolding.tryFold` catches `NonFatal` for expressions inside a conditional branch, tags them `FAILED_TO_EVALUATE` and leaves them unfolded, so a cast that throws can legitimately survive into the plan without ever being evaluated at runtime. Without that context, `catch { case NonFatal(_) => return None }` reads like swallowing an error. Worth also noting that `FAILED_TO_EVALUATE` would be the precise signal but is `private[sql]` and unusable from this package — otherwise someone will reasonably wonder why it is not used.



##########
spark/src/test/scala/org/apache/comet/serde/CometScalarFunctionSuite.scala:
##########
@@ -216,6 +219,41 @@ class CometScalarFunctionSuite extends CometTestBase {
     assertRejectReason(withContext, "CometScalarFunction", "evalContext")
   }
 
+  test("literal cast failure in an unvisited conditional branch does not fail planning") {
+    withTempPath { path =>
+      withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+        spark
+          .range(2)
+          .selectExpr("id", "CAST(id AS STRING) AS value")
+          .coalesce(1)
+          .write
+          .parquet(path.getCanonicalPath)
+      }
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+        withParquetTable(path.getCanonicalPath, "cast_branch_rows") {
+          val cast = "CAST(IF(id = 1, 'bad', value) AS INT)"
+          val masked = s"SELECT $cast AS parsed FROM cast_branch_rows LIMIT 1"
+          withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+            assert(sql(masked).collect().toSeq == Seq(Row(0)))
+          }
+          val (_, plan) = checkSparkAnswerAndFallbackReason(
+            masked,
+            "Literal cast requires Spark's conditional evaluation")
+          assert(collect(plan) { case scan: CometNativeScanExec => scan }.nonEmpty)
+          val (sparkError, cometError) =
+            checkSparkAnswerMaybeThrows(sql(s"SELECT $cast FROM cast_branch_rows"))
+          assert(sparkError.nonEmpty && cometError.nonEmpty)
+          val errors = Seq(sparkError.get, cometError.get).map { error =>
+            causeChain(error).collect { case e: SparkThrowable => e }.last
+          }
+          assert(errors.forall(_.getErrorClass == "CAST_INVALID_INPUT"))
+          assert(errors(0).getClass == errors(1).getClass)
+          assert(errors(0).getSqlState == errors(1).getSqlState)
+        }
+      }
+    }
+  }

Review Comment:
   Could you add a case asserting the fix is not over-corrected — that a *successful* literal cast (say `CAST('1' AS INT)` in the same branch position) still folds and stays native, with no fallback reason? #5623 paired its guard with exactly this kind of counterpart test, and without one there is nothing stopping the catch from being widened later.
   
   A non-ANSI case would also be worth having, or a note explaining why ANSI coverage is sufficient.



##########
spark/src/test/scala/org/apache/comet/serde/CometScalarFunctionSuite.scala:
##########
@@ -216,6 +219,41 @@ class CometScalarFunctionSuite extends CometTestBase {
     assertRejectReason(withContext, "CometScalarFunction", "evalContext")
   }
 
+  test("literal cast failure in an unvisited conditional branch does not fail planning") {
+    withTempPath { path =>
+      withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+        spark
+          .range(2)
+          .selectExpr("id", "CAST(id AS STRING) AS value")
+          .coalesce(1)
+          .write
+          .parquet(path.getCanonicalPath)
+      }
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+        withParquetTable(path.getCanonicalPath, "cast_branch_rows") {
+          val cast = "CAST(IF(id = 1, 'bad', value) AS INT)"
+          val masked = s"SELECT $cast AS parsed FROM cast_branch_rows LIMIT 1"
+          withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+            assert(sql(masked).collect().toSeq == Seq(Row(0)))

Review Comment:
   The test's validity rests on `ConstantFolding` refusing to fold `CAST('bad' AS INT)` because it sits in a conditional branch — but nothing here says so. If Spark's folding behavior ever changes, this test would keep passing while silently no longer covering the bug.
   
   A brief comment, or an assertion that the `Cast` is still present in the optimized plan, would pin the premise down.



##########
spark/src/test/scala/org/apache/comet/serde/CometScalarFunctionSuite.scala:
##########
@@ -216,6 +219,41 @@ class CometScalarFunctionSuite extends CometTestBase {
     assertRejectReason(withContext, "CometScalarFunction", "evalContext")
   }
 
+  test("literal cast failure in an unvisited conditional branch does not fail planning") {
+    withTempPath { path =>
+      withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+        spark
+          .range(2)
+          .selectExpr("id", "CAST(id AS STRING) AS value")
+          .coalesce(1)
+          .write
+          .parquet(path.getCanonicalPath)
+      }
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+        withParquetTable(path.getCanonicalPath, "cast_branch_rows") {
+          val cast = "CAST(IF(id = 1, 'bad', value) AS INT)"
+          val masked = s"SELECT $cast AS parsed FROM cast_branch_rows LIMIT 1"
+          withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+            assert(sql(masked).collect().toSeq == Seq(Row(0)))
+          }
+          val (_, plan) = checkSparkAnswerAndFallbackReason(
+            masked,
+            "Literal cast requires Spark's conditional evaluation")
+          assert(collect(plan) { case scan: CometNativeScanExec => scan }.nonEmpty)
+          val (sparkError, cometError) =
+            checkSparkAnswerMaybeThrows(sql(s"SELECT $cast FROM cast_branch_rows"))
+          assert(sparkError.nonEmpty && cometError.nonEmpty)
+          val errors = Seq(sparkError.get, cometError.get).map { error =>
+            causeChain(error).collect { case e: SparkThrowable => e }.last
+          }
+          assert(errors.forall(_.getErrorClass == "CAST_INVALID_INPUT"))
+          assert(errors(0).getClass == errors(1).getClass)
+          assert(errors(0).getSqlState == errors(1).getSqlState)

Review Comment:
   `CometTestBase.checkSparkError(df, errorClass)` already does all of this, and is stricter in two ways worth keeping:
   
   - it asserts no `CometNativeException` appears in the cause chain, so the test cannot pass on an error that surfaced from native code with a coincidentally matching class;
   - it uses `lastOption.getOrElse(fail(...))`, whereas `.last` here throws `UnsupportedOperationException` if the chain contains no `SparkThrowable`, turning a meaningful assertion failure into a confusing one.
   
   The whole block collapses to:
   
   ```scala
   checkSparkError(sql(s"SELECT $cast FROM cast_branch_rows"), "CAST_INVALID_INPUT")
   ```
   
   which also reads better than the `errors(0)` / `errors(1)` indexing.
   
   (Unrelated to this PR: `getErrorClass` is deprecated in favor of `getCondition()`, but the shared helper still uses it, so that is better changed there on its own.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: defer throwing literal casts to Spark at runtime [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6035:
URL: https://github.com/apache/datafusion-comet/pull/6035#discussion_r4178682477


##########
spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:
##########
@@ -109,7 +111,15 @@ object CometCast
     val cometEvalMode = evalMode(cast)
     cast.child match {
       case _: Literal =>
-        exprToProtoInternal(Literal.create(cast.eval(), cast.dataType), inputs, binding)
+        val value =
+          try {
+            cast.eval()
+          } catch {
+            case NonFatal(_) =>
+              withFallbackReason(cast, "Literal cast requires Spark's conditional evaluation")
+              return None
+          }

Review Comment:
   Added a shared explanatory reason and the ConstantFolding/FAILED_TO_EVALUATE comment. I kept full projection fallback deliberately: batch codegen dispatch can evaluate the bad cast on a row Spark never reaches after LIMIT. The regression covers both dispatcher settings and passes; successful/non-ANSI controls stay native.



##########
spark/src/test/scala/org/apache/comet/serde/CometScalarFunctionSuite.scala:
##########
@@ -216,6 +219,41 @@ class CometScalarFunctionSuite extends CometTestBase {
     assertRejectReason(withContext, "CometScalarFunction", "evalContext")
   }
 
+  test("literal cast failure in an unvisited conditional branch does not fail planning") {

Review Comment:
   Moved the end-to-end regression into CometNativeCastSuite; both focused cast tests pass on Spark 4.1.



##########
spark/src/test/scala/org/apache/comet/serde/CometScalarFunctionSuite.scala:
##########
@@ -216,6 +219,41 @@ class CometScalarFunctionSuite extends CometTestBase {
     assertRejectReason(withContext, "CometScalarFunction", "evalContext")
   }
 
+  test("literal cast failure in an unvisited conditional branch does not fail planning") {
+    withTempPath { path =>
+      withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+        spark
+          .range(2)
+          .selectExpr("id", "CAST(id AS STRING) AS value")
+          .coalesce(1)
+          .write
+          .parquet(path.getCanonicalPath)
+      }
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+        withParquetTable(path.getCanonicalPath, "cast_branch_rows") {
+          val cast = "CAST(IF(id = 1, 'bad', value) AS INT)"
+          val masked = s"SELECT $cast AS parsed FROM cast_branch_rows LIMIT 1"
+          withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+            assert(sql(masked).collect().toSeq == Seq(Row(0)))

Review Comment:
   The regression now asserts the optimized plan still contains the cast of the invalid literal before executing the query, pinning the ConstantFolding premise.



##########
spark/src/test/scala/org/apache/comet/serde/CometScalarFunctionSuite.scala:
##########
@@ -216,6 +219,41 @@ class CometScalarFunctionSuite extends CometTestBase {
     assertRejectReason(withContext, "CometScalarFunction", "evalContext")
   }
 
+  test("literal cast failure in an unvisited conditional branch does not fail planning") {
+    withTempPath { path =>
+      withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+        spark
+          .range(2)
+          .selectExpr("id", "CAST(id AS STRING) AS value")
+          .coalesce(1)
+          .write
+          .parquet(path.getCanonicalPath)
+      }
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+        withParquetTable(path.getCanonicalPath, "cast_branch_rows") {
+          val cast = "CAST(IF(id = 1, 'bad', value) AS INT)"
+          val masked = s"SELECT $cast AS parsed FROM cast_branch_rows LIMIT 1"
+          withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+            assert(sql(masked).collect().toSeq == Seq(Row(0)))
+          }
+          val (_, plan) = checkSparkAnswerAndFallbackReason(
+            masked,
+            "Literal cast requires Spark's conditional evaluation")
+          assert(collect(plan) { case scan: CometNativeScanExec => scan }.nonEmpty)
+          val (sparkError, cometError) =
+            checkSparkAnswerMaybeThrows(sql(s"SELECT $cast FROM cast_branch_rows"))
+          assert(sparkError.nonEmpty && cometError.nonEmpty)
+          val errors = Seq(sparkError.get, cometError.get).map { error =>
+            causeChain(error).collect { case e: SparkThrowable => e }.last
+          }
+          assert(errors.forall(_.getErrorClass == "CAST_INVALID_INPUT"))
+          assert(errors(0).getClass == errors(1).getClass)
+          assert(errors(0).getSqlState == errors(1).getSqlState)

Review Comment:
   Switched to checkSparkError. Its new optional checkNative argument defaults to true; this intentional projection-fallback case disables only that plan assertion and retains the shared exception, error-class, SQLSTATE and cause-chain checks.



##########
spark/src/test/scala/org/apache/comet/serde/CometScalarFunctionSuite.scala:
##########
@@ -216,6 +219,41 @@ class CometScalarFunctionSuite extends CometTestBase {
     assertRejectReason(withContext, "CometScalarFunction", "evalContext")
   }
 
+  test("literal cast failure in an unvisited conditional branch does not fail planning") {
+    withTempPath { path =>
+      withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+        spark
+          .range(2)
+          .selectExpr("id", "CAST(id AS STRING) AS value")
+          .coalesce(1)
+          .write
+          .parquet(path.getCanonicalPath)
+      }
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+        withParquetTable(path.getCanonicalPath, "cast_branch_rows") {
+          val cast = "CAST(IF(id = 1, 'bad', value) AS INT)"
+          val masked = s"SELECT $cast AS parsed FROM cast_branch_rows LIMIT 1"
+          withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+            assert(sql(masked).collect().toSeq == Seq(Row(0)))
+          }
+          val (_, plan) = checkSparkAnswerAndFallbackReason(
+            masked,
+            "Literal cast requires Spark's conditional evaluation")
+          assert(collect(plan) { case scan: CometNativeScanExec => scan }.nonEmpty)
+          val (sparkError, cometError) =
+            checkSparkAnswerMaybeThrows(sql(s"SELECT $cast FROM cast_branch_rows"))
+          assert(sparkError.nonEmpty && cometError.nonEmpty)
+          val errors = Seq(sparkError.get, cometError.get).map { error =>
+            causeChain(error).collect { case e: SparkThrowable => e }.last
+          }
+          assert(errors.forall(_.getErrorClass == "CAST_INVALID_INPUT"))
+          assert(errors(0).getClass == errors(1).getClass)
+          assert(errors(0).getSqlState == errors(1).getSqlState)
+        }
+      }
+    }
+  }

Review Comment:
   Added Parquet-backed successful literal-cast controls in ANSI and non-ANSI modes plus an invalid non-ANSI control. All assert native CometProject execution and absence of the fallback reason; the focused tests pass.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: defer throwing literal casts to Spark at runtime [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #6035:
URL: https://github.com/apache/datafusion-comet/pull/6035#discussion_r4052397023


##########
spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:
##########
@@ -109,7 +111,15 @@ object CometCast
     val cometEvalMode = evalMode(cast)
     cast.child match {
       case _: Literal =>
-        exprToProtoInternal(Literal.create(cast.eval(), cast.dataType), inputs, binding)
+        val value =
+          try {
+            cast.eval()
+          } catch {
+            case NonFatal(_) =>
+              withFallbackReason(cast, "Literal cast requires Spark's conditional evaluation")
+              return None
+          }

Review Comment:
   Agree with @viirya would be nice to have more reasonable fall back message



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: cap native projection input before collect limits [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #6035:
URL: https://github.com/apache/datafusion-comet/pull/6035#issuecomment-6282991945

   @viirya, I revised the PR to address your [latest review](https://github.com/apache/datafusion-comet/pull/6035#pullrequestreview-5576270147). You are right that only the cast is dispatched and that the native conditional masks its input. The whole-projection fallback is removed. `CometCast.scala`, `cast_literal_ansi.sql`, and the shared error helper now match main, preserving #6826's dispatch and native-execution guarantees.
   
   There is still a separate LIMIT case. A minimal reproducer uses ANSI mode and Parquet rows `(id, value) = (0, '0'), (1, '1')` in the same batch:
   
   ```sql
   SELECT CAST(IF(id = 1, 'bad', value) AS INT) AS parsed
   FROM cast_branch_rows
   LIMIT 1
   ```
   
   Spark returns `0`. On the PR's main base (`80b518173`), Comet raises `CAST_INVALID_INPUT`. Row 1 actually satisfies the IF condition, so branch masking correctly selects it. The problem is that the projection evaluates that row before the outer LIMIT can stop consumption.
   
   The PR now caps the input of an immediately adjacent native projection:
   
   ```text
   CollectLimit(n, offset)
     CometProject
       CometLocalLimit(n)
         child
   ```
   
   The outer limit still combines partitions and drops OFFSET rows. Column selections and aliases skip the extra cap to preserve filter column pruning. Other plan shapes are outside this PR.
   
   At `1dc896732`, 116 local Spark 4.0 tests passed, including native projection/cast-dispatch assertions, demanded errors, OFFSET, multiple batches and partitions, AQE, and pruning coverage. Spotless and Scalastyle also passed. Hosted CI, including the requested Spark 4.1 SQL coverage, is still pending.
   
   I also replied to the three inline comments about assertions, folding coverage, and `checkNative`. Could you take another look at the revised scope?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: cap native projection input before collect limits [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6035:
URL: https://github.com/apache/datafusion-comet/pull/6035#discussion_r4340653485


##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -724,6 +724,54 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
     }
   }
 
+  /**
+   * Spark's collect limit consumes at most `limit` rows per partition, but a native projection
+   * evaluates its whole input batch before that limit can stop reading. Limit the projection's
+   * input first so an expression on an unconsumed row cannot raise an error. Keep the outer
+   * limit: it still combines partitions and drops OFFSET rows, which Spark evaluates before
+   * dropping.
+   *
+   * Only handle an immediately adjacent native projection. A filter or join between the limit and
+   * projection can require more input rows, and a blocking operator must consume its input.
+   */

Review Comment:
   Confirmed: the non-collect limit plan has the same preexisting defect. I reproduced `CometGlobalLimit -> exchange -> CometLocalLimit -> CometProject` on Spark 4.0.4: Spark returns `0`, while Comet raises `CAST_INVALID_INPUT`. The native local limit slices the projection's output after the projection has evaluated the batch, so it does not protect that projection's input.
   
   Filed #7147 with that reproducer and the throwing-filter case, under the broader tracking issue #6006. The rule's scaladoc and compatibility guide now link it. The issue also tracks direct GlobalLimit coverage and already-ordered `TakeOrderedAndProject` inputs for investigation. It distinguishes these from blocking operators and `CollectTail`, where Spark itself may need to evaluate discarded rows. This PR keeps the adjacent collect-limit fix scoped as discussed.
   
   Updated in `eea8a6b00`. The focused Spark 4.0 run passed all 116 tests.



##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -724,6 +724,54 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
     }
   }
 
+  /**
+   * Spark's collect limit consumes at most `limit` rows per partition, but a native projection
+   * evaluates its whole input batch before that limit can stop reading. Limit the projection's
+   * input first so an expression on an unconsumed row cannot raise an error. Keep the outer
+   * limit: it still combines partitions and drops OFFSET rows, which Spark evaluates before
+   * dropping.
+   *
+   * Only handle an immediately adjacent native projection. A filter or join between the limit and
+   * projection can require more input rows, and a blocking operator must consume its input.
+   */
+  private def limitCollectProjectionInput(op: SparkPlan): SparkPlan = op match {
+    case limit: CollectLimitExec if limit.limit >= 0 =>
+      limit.child match {
+        case project: CometProjectExec =>
+          // Selecting or renaming columns cannot throw. Keep Project -> Filter adjacent so
+          // native planning can prune columns used only by the filter's predicate.
+          val onlyColumns = project.projectList.forall {
+            case _: Attribute | Alias(_: Attribute, _) => true
+            case _ => false
+          }
+          if (onlyColumns) return op
+          project.child match {
+            case cap: CometLocalLimitExec if cap.limit <= limit.limit => return op
+            case _ =>
+          }
+          val bounded = for {
+            cap <- convertToComet(LocalLimitExec(limit.limit, project.child), CometLocalLimitExec)

Review Comment:
   Added the comment beside the synthetic conversion. It explains that `convertToComet` deliberately applies the normal configuration and serde checks, and that a declined synthetic limit is discarded together with its fallback tags while the original plan is retained.
   
   Updated in `eea8a6b00`. The focused Spark 4.0 run passed all 116 tests.



##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -724,6 +724,54 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
     }
   }
 
+  /**
+   * Spark's collect limit consumes at most `limit` rows per partition, but a native projection
+   * evaluates its whole input batch before that limit can stop reading. Limit the projection's
+   * input first so an expression on an unconsumed row cannot raise an error. Keep the outer
+   * limit: it still combines partitions and drops OFFSET rows, which Spark evaluates before
+   * dropping.
+   *
+   * Only handle an immediately adjacent native projection. A filter or join between the limit and
+   * projection can require more input rows, and a blocking operator must consume its input.
+   */
+  private def limitCollectProjectionInput(op: SparkPlan): SparkPlan = op match {
+    case limit: CollectLimitExec if limit.limit >= 0 =>
+      limit.child match {
+        case project: CometProjectExec =>
+          // Selecting or renaming columns cannot throw. Keep Project -> Filter adjacent so
+          // native planning can prune columns used only by the filter's predicate.
+          val onlyColumns = project.projectList.forall {
+            case _: Attribute | Alias(_: Attribute, _) => true
+            case _ => false
+          }
+          if (onlyColumns) return op

Review Comment:
   Replaced both early returns with `if (onlyColumns || alreadyCapped) op else { ... }`. The existing `cap.limit <= limit.limit` check and column-pruning guard are preserved.
   
   Updated in `eea8a6b00`. The focused Spark 4.0 run passed all 116 tests.



##########
spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala:
##########
@@ -65,6 +66,112 @@ class CometNativeCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
 
   import testImplicits._
 
+  test("LIMIT skips a failing conditional literal cast and retains Comet projection") {
+    withTempPath { path =>
+      withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+        spark
+          .range(5)
+          .selectExpr("id", "CAST(id AS STRING) AS value")
+          .coalesce(1)
+          .write
+          .parquet(path.getCanonicalPath)
+      }
+      withSQLConf(
+        SQLConf.ANSI_ENABLED.key -> "true",
+        CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key -> "true",
+        CometConf.COMET_BATCH_SIZE.key -> "8192") {
+        withParquetTable(path.getCanonicalPath, "cast_branch_rows") {
+          // Both rows share a batch. The native projection must receive only row 0: masking
+          // the cast by its IF condition alone would still evaluate the failing row 1.
+          val cast = "CAST(IF(id = 1, 'bad', value) AS INT)"
+          val masked = s"SELECT $cast AS parsed FROM cast_branch_rows LIMIT 1"
+          // ConstantFolding must leave the throwing literal cast inside the conditional.
+          val badCasts = sql(masked).queryExecution.optimizedPlan
+            .flatMap(_.expressions)
+            .flatMap(_.collect {
+              case c: Cast
+                  if c.child.isInstanceOf[Literal] &&
+                    Option(c.child.asInstanceOf[Literal].value).exists(_.toString == "bad") =>
+                c
+            })
+          assert(badCasts.nonEmpty)
+          val (_, plan) = checkSparkAnswerAndOperator(
+            sql(masked),
+            includeClasses = Seq(classOf[CometProjectExec]))
+          assert(new ExtendedExplainInfo().getCodegenDispatchExpressions(plan).contains("cast"))
+          checkSparkError(
+            sql(s"SELECT $cast FROM cast_branch_rows LIMIT 2"),
+            "CAST_INVALID_INPUT")
+
+          // CollectLimit evaluates the prefix before dropping OFFSET rows. Moving an offset
+          // below the projection would incorrectly hide this error in the skipped first row.
+          checkSparkError(
+            sql(
+              "SELECT CAST(IF(id = 0, 'bad', value) AS INT) " +
+                "FROM cast_branch_rows LIMIT 1 OFFSET 1"),
+            "CAST_INVALID_INPUT")
+
+          withSQLConf(CometConf.COMET_BATCH_SIZE.key -> "2") {

Review Comment:
   Kept both tests unchanged: the batch-size-2 case checks the cumulative cap, and the per-partition case still uses `executeColumnar` so the second partition is exercised.
   
   Updated in `eea8a6b00`. The focused Spark 4.0 run passed all 116 tests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: cap native projection input before collect limits [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #6035:
URL: https://github.com/apache/datafusion-comet/pull/6035#issuecomment-6323894106

   @andygrove @viirya, addressed the latest review in `eea8a6b00`.
   
   The [compatibility guide](https://github.com/apache/datafusion-comet/blob/eea8a6b0004e6a4c2a076216629246000a5b72ca/docs/source/user-guide/latest/compatibility/index.md#ansi-errors-from-rows-skipped-by-limit) now explains the ANSI error difference and includes Andy's throwing-filter example. Filed #7147 under the broader #6006 with the reproduced filter and non-collect LocalLimit/GlobalLimit plan cases, their physical plans, and the remaining ordered-input `TakeOrderedAndProject` investigation. These are preexisting gaps left for separate fixes. The issue also distinguishes discarded rows that Spark legitimately evaluates, including blocking operations and `CollectTail`.
   
   The rule's scaladoc links the issue. Added the requested synthetic-node config/serde and fallback-tag explanation, combined the two early exits into one conditional, and retained the cross-batch and per-partition tests unchanged. I replied to all four new inline threads and resolved those plus the three older addressed threads.
   
   Validation on this commit: native debug build, all 116 focused Spark 4.0 tests, Spotless, Scalastyle, repository-wide Markdown Prettier, and `git diff --check` passed. All nine Spark 4.1 SQL jobs passed on the previous head `1dc896732` ([run](https://github.com/apache/datafusion-comet/actions/runs/39365819149)); the `run-spark-4.1-tests` label remains enabled, and [CI for this new commit](https://github.com/apache/datafusion-comet/actions/runs/39655951685) is queued.
   
   Could you re-review the update? The earlier changes-requested review is still active, so it needs an updated review verdict before merging.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: cap native projection input before collect limits [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao merged PR #6035:
URL: https://github.com/apache/datafusion-comet/pull/6035


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: cap native projection input before collect limits [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6035:
URL: https://github.com/apache/datafusion-comet/pull/6035#discussion_r4353919707


##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -724,6 +724,64 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
     }
   }
 
+  /**
+   * Spark's collect limit consumes at most `limit` rows per partition, but a native projection
+   * evaluates its whole input batch before that limit can stop reading. Limit the projection's
+   * input first so an expression on an unconsumed row cannot raise an error. Keep the outer
+   * limit: it still combines partitions and drops OFFSET rows, which Spark evaluates before
+   * dropping.
+   *
+   * Only handle an immediately adjacent native projection. A filter or join between the limit and
+   * projection can require more input rows, and a blocking operator must consume its input.
+   * Non-collect LocalLimit/GlobalLimit plans can still evaluate a projection before limiting its
+   * output. That preexisting gap and throwing filter predicates are tracked separately in
+   * https://github.com/apache/datafusion-comet/issues/7147.

Review Comment:
   Agreed. #7147 already records that native planning wraps the projection in a limit, allowing the projection to evaluate the whole batch before its output is sliced. I added an explicit follow-up item to carry that mechanism into the rule's scaladoc if the fix extends `limitCollectProjectionInput`. Keeping that change with the follow-up as suggested.



##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -724,6 +724,64 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
     }
   }
 
+  /**
+   * Spark's collect limit consumes at most `limit` rows per partition, but a native projection
+   * evaluates its whole input batch before that limit can stop reading. Limit the projection's
+   * input first so an expression on an unconsumed row cannot raise an error. Keep the outer
+   * limit: it still combines partitions and drops OFFSET rows, which Spark evaluates before
+   * dropping.
+   *
+   * Only handle an immediately adjacent native projection. A filter or join between the limit and
+   * projection can require more input rows, and a blocking operator must consume its input.
+   * Non-collect LocalLimit/GlobalLimit plans can still evaluate a projection before limiting its
+   * output. That preexisting gap and throwing filter predicates are tracked separately in
+   * https://github.com/apache/datafusion-comet/issues/7147.
+   */
+  private def limitCollectProjectionInput(op: SparkPlan): SparkPlan = op match {
+    case limit: CollectLimitExec if limit.limit >= 0 =>
+      limit.child match {
+        case project: CometProjectExec =>
+          // Selecting or renaming columns cannot throw. Keep Project -> Filter adjacent so
+          // native planning can prune columns used only by the filter's predicate.
+          val onlyColumns = project.projectList.forall {
+            case _: Attribute | Alias(_: Attribute, _) => true
+            case _ => false
+          }
+          val alreadyCapped = project.child match {
+            case cap: CometLocalLimitExec => cap.limit <= limit.limit
+            case _ => false
+          }
+          if (onlyColumns || alreadyCapped) {
+            op

Review Comment:
   Thanks for confirming. Keeping the combined conditional and the existing `cap.limit <= limit.limit` guard unchanged.



##########
docs/source/user-guide/latest/compatibility/index.md:
##########
@@ -106,6 +106,32 @@ Separately, Comet's native Parquet scan currently rejects string columns whose s
 valid UTF-8 rather than reading them like Spark
 ([#4121](https://github.com/apache/datafusion-comet/issues/4121)).
 
+## ANSI errors from rows skipped by LIMIT
+
+Native operators evaluate batches of rows. With `spark.sql.ansi.enabled=true`, some `LIMIT`
+plan shapes can therefore raise an expression error from a row that Spark stops before evaluating.
+For example, with a Parquet batch read in the order `(id, value) = (0, '0'), (1, '1')`:
+
+```sql
+SELECT value FROM t
+WHERE CAST(IF(id = 1, 'bad', value) AS INT) >= 0
+LIMIT 1
+```
+
+Spark returns `'0'` after the first row matches. Comet can evaluate the filter predicate on the
+second row in the same batch and raise `CAST_INVALID_INPUT`. SQL without `ORDER BY` does not
+guarantee this input order; the example describes a physical scan order that exposes the difference.
+
+Comet caps projection input when a collect limit directly consumes the projection. This protects
+that projection's expressions, but does not bound evaluation inside its child operators. Throwing
+filter predicates and projections beneath non-collect `LocalLimit` / `GlobalLimit` plans, such as
+a `LIMIT` inside a subquery, remain affected. These existing limitations and the investigation of
+already-ordered `TakeOrderedAndProject` inputs are tracked in
+[#7147](https://github.com/apache/datafusion-comet/issues/7147).
+
+An error on a row absent from the final result is not always a mismatch: Spark can also need to
+evaluate that row, for example to sort the input or process an `OFFSET` prefix.

Review Comment:
   Thanks. Keeping both the physical-order qualification and the distinction between a discarded row and a row Spark never evaluates.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: defer throwing literal casts to Spark at runtime [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #6035:
URL: https://github.com/apache/datafusion-comet/pull/6035#issuecomment-5799510899

   Thanks! Agree it is not ideal to fallback to Spark fully for this scenario. I should have checked before Codex-ing this PR lol. Let me revise it and then ask for another round of reviews.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: cap native projection input before collect limits [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6035:
URL: https://github.com/apache/datafusion-comet/pull/6035#discussion_r4321302685


##########
spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala:
##########
@@ -65,6 +66,70 @@ class CometNativeCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
 
   import testImplicits._
 
+  test("literal cast failure in an unvisited conditional branch does not fail planning") {
+    withTempPath { path =>
+      withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+        spark
+          .range(2)
+          .selectExpr("id", "CAST(id AS STRING) AS value")
+          .coalesce(1)
+          .write
+          .parquet(path.getCanonicalPath)
+      }
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+        withParquetTable(path.getCanonicalPath, "cast_branch_rows") {
+          val cast = "CAST(IF(id = 1, 'bad', value) AS INT)"
+          val masked = s"SELECT $cast AS parsed FROM cast_branch_rows LIMIT 1"
+          withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+            assert(sql(masked).collect().toSeq == Seq(Row(0)))
+          }
+          // ConstantFolding must leave the throwing literal cast inside the conditional.
+          val badCasts = sql(masked).queryExecution.optimizedPlan
+            .flatMap(_.expressions)
+            .flatMap(_.collect {
+              case c: Cast
+                  if c.child.isInstanceOf[Literal] &&
+                    Option(c.child.asInstanceOf[Literal].value).exists(_.toString == "bad") =>
+                c
+            })
+          assert(badCasts.nonEmpty)
+          val (_, plan) =
+            checkSparkAnswerAndFallbackReason(masked, CometCast.literalCastConditionalEvalReason)
+          assert(collect(plan) { case scan: CometNativeScanExec => scan }.nonEmpty)
+          checkSparkError(
+            sql(s"SELECT $cast FROM cast_branch_rows"),
+            "CAST_INVALID_INPUT",
+            checkNative = false)
+        }
+      }
+    }
+  }
+
+  test("successful literal casts and non-ANSI invalid casts retain native projection") {
+    withParquetTable(Seq(Tuple1(0), Tuple1(1)), "cast_branch_controls") {
+      for (ansi <- Seq("true", "false")) {
+        withSQLConf(SQLConf.ANSI_ENABLED.key -> ansi) {
+          val query = "SELECT CAST(IF(_1 = 1, '1', CAST(_1 AS STRING)) AS INT) " +
+            "FROM cast_branch_controls"
+          val (_, plan) = checkSparkAnswerAndOperator(sql(query))
+          assert(collect(plan) { case project: CometProjectExec => project }.nonEmpty)
+          assert(
+            !plan.exists(_.getTagValue(CometExplainInfo.FALLBACK_REASONS)
+              .exists(_.contains(CometCast.literalCastConditionalEvalReason))))

Review Comment:
   Removed the fallback-reason checks and redundant projection checks from these controls. They now use `checkSparkAnswerAndOperator` for Spark parity and native execution. The revised PR also removes the whole-projection fallback, so there is no new literal-cast fallback reason to assert against. Fixed in the current head, `1dc896732`.



##########
spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala:
##########
@@ -65,6 +66,70 @@ class CometNativeCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
 
   import testImplicits._
 
+  test("literal cast failure in an unvisited conditional branch does not fail planning") {
+    withTempPath { path =>
+      withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+        spark
+          .range(2)
+          .selectExpr("id", "CAST(id AS STRING) AS value")
+          .coalesce(1)
+          .write
+          .parquet(path.getCanonicalPath)
+      }
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+        withParquetTable(path.getCanonicalPath, "cast_branch_rows") {
+          val cast = "CAST(IF(id = 1, 'bad', value) AS INT)"
+          val masked = s"SELECT $cast AS parsed FROM cast_branch_rows LIMIT 1"
+          withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+            assert(sql(masked).collect().toSeq == Seq(Row(0)))
+          }
+          // ConstantFolding must leave the throwing literal cast inside the conditional.
+          val badCasts = sql(masked).queryExecution.optimizedPlan
+            .flatMap(_.expressions)
+            .flatMap(_.collect {
+              case c: Cast
+                  if c.child.isInstanceOf[Literal] &&
+                    Option(c.child.asInstanceOf[Literal].value).exists(_.toString == "bad") =>
+                c
+            })
+          assert(badCasts.nonEmpty)
+          val (_, plan) =
+            checkSparkAnswerAndFallbackReason(masked, CometCast.literalCastConditionalEvalReason)
+          assert(collect(plan) { case scan: CometNativeScanExec => scan }.nonEmpty)
+          checkSparkError(
+            sql(s"SELECT $cast FROM cast_branch_rows"),
+            "CAST_INVALID_INPUT",
+            checkNative = false)
+        }
+      }
+    }
+  }
+
+  test("successful literal casts and non-ANSI invalid casts retain native projection") {

Review Comment:
   Added direct serialization assertions in `successful and non-ANSI literal casts fold and retain Comet projection`: `QueryPlanSerde.exprToProto` must produce literal `1` for the successful cast in both ANSI modes, and literal `null` for the invalid non-ANSI cast. Calling the serializer directly prevents Spark's ConstantFolding from hiding Comet's folding behavior. The Parquet queries still check Spark parity and native execution. These tests passed on Spark 4.0 at `1dc896732`.



##########
spark/src/test/scala/org/apache/spark/sql/CometTestBase.scala:
##########
@@ -452,11 +452,14 @@ abstract class CometTestBase
     }
   }
 
-  /** Checks native execution and Spark exception type, error class and SQLSTATE parity. */
+  /** Checks Spark exception parity and, by default, native execution. */

Review Comment:
   The revised approach keeps the projection native, so the `checkNative` parameter is no longer needed and I removed it. `CometTestBase.scala` now matches main again. The new demanded-error cases use the original strict `checkSparkError` helper, including its native-plan and exception/cause-chain checks. This supersedes my earlier reply about opting out of the native assertion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: cap native projection input before collect limits [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #6035:
URL: https://github.com/apache/datafusion-comet/pull/6035#issuecomment-6377231866

   Thanks for the review! Added to merge queue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: defer throwing literal casts to Spark at runtime [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #6035:
URL: https://github.com/apache/datafusion-comet/pull/6035#discussion_r4288566394


##########
spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala:
##########
@@ -65,6 +66,70 @@ class CometNativeCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
 
   import testImplicits._
 
+  test("literal cast failure in an unvisited conditional branch does not fail planning") {
+    withTempPath { path =>
+      withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+        spark
+          .range(2)
+          .selectExpr("id", "CAST(id AS STRING) AS value")
+          .coalesce(1)
+          .write
+          .parquet(path.getCanonicalPath)
+      }
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+        withParquetTable(path.getCanonicalPath, "cast_branch_rows") {
+          val cast = "CAST(IF(id = 1, 'bad', value) AS INT)"
+          val masked = s"SELECT $cast AS parsed FROM cast_branch_rows LIMIT 1"
+          withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+            assert(sql(masked).collect().toSeq == Seq(Row(0)))
+          }
+          // ConstantFolding must leave the throwing literal cast inside the conditional.
+          val badCasts = sql(masked).queryExecution.optimizedPlan
+            .flatMap(_.expressions)
+            .flatMap(_.collect {
+              case c: Cast
+                  if c.child.isInstanceOf[Literal] &&
+                    Option(c.child.asInstanceOf[Literal].value).exists(_.toString == "bad") =>
+                c
+            })
+          assert(badCasts.nonEmpty)
+          val (_, plan) =
+            checkSparkAnswerAndFallbackReason(masked, CometCast.literalCastConditionalEvalReason)
+          assert(collect(plan) { case scan: CometNativeScanExec => scan }.nonEmpty)
+          checkSparkError(
+            sql(s"SELECT $cast FROM cast_branch_rows"),
+            "CAST_INVALID_INPUT",
+            checkNative = false)
+        }
+      }
+    }
+  }
+
+  test("successful literal casts and non-ANSI invalid casts retain native projection") {
+    withParquetTable(Seq(Tuple1(0), Tuple1(1)), "cast_branch_controls") {
+      for (ansi <- Seq("true", "false")) {
+        withSQLConf(SQLConf.ANSI_ENABLED.key -> ansi) {
+          val query = "SELECT CAST(IF(_1 = 1, '1', CAST(_1 AS STRING)) AS INT) " +
+            "FROM cast_branch_controls"
+          val (_, plan) = checkSparkAnswerAndOperator(sql(query))
+          assert(collect(plan) { case project: CometProjectExec => project }.nonEmpty)
+          assert(
+            !plan.exists(_.getTagValue(CometExplainInfo.FALLBACK_REASONS)
+              .exists(_.contains(CometCast.literalCastConditionalEvalReason))))

Review Comment:
   This assertion cannot fail, so it is not testing what it looks like it is testing:
   
   ```scala
   assert(
     !plan.exists(_.getTagValue(CometExplainInfo.FALLBACK_REASONS)
       .exists(_.contains(CometCast.literalCastConditionalEvalReason))))
   ```
   
   `withFallbackReason(cast, ...)` tags the `Cast` **expression**, not a plan node. An expression-level reason only reaches the operator through `CometExecRule.rollUpFallbackReasons`, which runs on just one path:
   
   ```scala
   val converted = tryConvertToComet(op, handler)
   if (converted.isEmpty) {
     rollUpFallbackReasons(op)
     reportUnexplainedFallback(op)
   }
   ```
   
   In this control the conversion succeeds — that is the point of the test — so the roll-up never runs and the plan node never carries the tag. The assertion passes whether or not the reason was recorded on the expression.
   
   Switching to `ExtendedExplainInfo.getFallbackReasons(plan)` would not help either; it reads the same plan-node tags and depends on the same roll-up.
   
   It is also redundant. `checkSparkAnswerAndOperator` already asserts through `checkCometOperators` / `findFirstNonCometOperator` that every operator in the plan is a Comet operator, which is a stronger statement than either this assertion or the `CometProjectExec` collect on the line above. Dropping both lines in each of the three control blocks would leave a clearer and strictly stronger guarantee:
   
   ```scala
   val (_, plan) = checkSparkAnswerAndOperator(sql(query))
   ```
   
   If you would rather assert the reason's absence directly, it has to be read off the expression tree rather than the plan node — e.g. collecting the `Cast` nodes out of `plan.expressions` and checking their own tags.



##########
spark/src/test/scala/org/apache/spark/sql/CometTestBase.scala:
##########
@@ -452,11 +452,14 @@ abstract class CometTestBase
     }
   }
 
-  /** Checks native execution and Spark exception type, error class and SQLSTATE parity. */
+  /** Checks Spark exception parity and, by default, native execution. */

Review Comment:
   Adding `checkNative` with a `true` default is the right shape here — every existing caller keeps the stricter behavior, and only this deliberate-fallback case opts out while retaining the exception, error-class, SQLSTATE and cause-chain parity checks.
   
   Worth considering a short note in the scaladoc that `checkNative = false` is for cases where the fallback is the expected outcome, so the escape hatch does not get reached for merely inconvenient plan assertions later.



##########
spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala:
##########
@@ -65,6 +66,70 @@ class CometNativeCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
 
   import testImplicits._
 
+  test("literal cast failure in an unvisited conditional branch does not fail planning") {
+    withTempPath { path =>
+      withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+        spark
+          .range(2)
+          .selectExpr("id", "CAST(id AS STRING) AS value")
+          .coalesce(1)
+          .write
+          .parquet(path.getCanonicalPath)
+      }
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+        withParquetTable(path.getCanonicalPath, "cast_branch_rows") {
+          val cast = "CAST(IF(id = 1, 'bad', value) AS INT)"
+          val masked = s"SELECT $cast AS parsed FROM cast_branch_rows LIMIT 1"
+          withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+            assert(sql(masked).collect().toSeq == Seq(Row(0)))
+          }
+          // ConstantFolding must leave the throwing literal cast inside the conditional.
+          val badCasts = sql(masked).queryExecution.optimizedPlan
+            .flatMap(_.expressions)
+            .flatMap(_.collect {
+              case c: Cast
+                  if c.child.isInstanceOf[Literal] &&
+                    Option(c.child.asInstanceOf[Literal].value).exists(_.toString == "bad") =>
+                c
+            })
+          assert(badCasts.nonEmpty)
+          val (_, plan) =
+            checkSparkAnswerAndFallbackReason(masked, CometCast.literalCastConditionalEvalReason)
+          assert(collect(plan) { case scan: CometNativeScanExec => scan }.nonEmpty)
+          checkSparkError(
+            sql(s"SELECT $cast FROM cast_branch_rows"),
+            "CAST_INVALID_INPUT",
+            checkNative = false)
+        }
+      }
+    }
+  }
+
+  test("successful literal casts and non-ANSI invalid casts retain native projection") {

Review Comment:
   Minor: the test name says "retain native projection", which is what the assertions check, but the more valuable property here is that a *successful* literal cast still takes the folding path rather than the new fallback — that is the over-correction guard. Since the fallback-reason assertion above does not actually establish that (see the other comment), consider either asserting the folded literal appears in the serialized projection, or renaming to reflect what is really covered, so a future reader does not over-trust it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: cap native projection input before collect limits [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #6035:
URL: https://github.com/apache/datafusion-comet/pull/6035#discussion_r4342677938


##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -724,6 +724,64 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
     }
   }
 
+  /**
+   * Spark's collect limit consumes at most `limit` rows per partition, but a native projection
+   * evaluates its whole input batch before that limit can stop reading. Limit the projection's
+   * input first so an expression on an unconsumed row cannot raise an error. Keep the outer
+   * limit: it still combines partitions and drops OFFSET rows, which Spark evaluates before
+   * dropping.
+   *
+   * Only handle an immediately adjacent native projection. A filter or join between the limit and
+   * projection can require more input rows, and a blocking operator must consume its input.
+   * Non-collect LocalLimit/GlobalLimit plans can still evaluate a projection before limiting its
+   * output. That preexisting gap and throwing filter predicates are tracked separately in
+   * https://github.com/apache/datafusion-comet/issues/7147.

Review Comment:
   This is the right place for it, and the wording does the job: it says the gap exists, names the shape, and points at the issue rather than leaving a future reader to re-derive what I had to go and check.
   
   One thought for #7147 rather than this PR: the mechanism you found — that the native local limit slices the projection's output, so it cannot bound that projection's input — is the part that makes the non-collect shape non-obvious, and it isn't captured in this sentence. If #7147's fix ends up extending this rule, that sentence is where the next person will start.



##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -724,6 +724,64 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
     }
   }
 
+  /**
+   * Spark's collect limit consumes at most `limit` rows per partition, but a native projection
+   * evaluates its whole input batch before that limit can stop reading. Limit the projection's
+   * input first so an expression on an unconsumed row cannot raise an error. Keep the outer
+   * limit: it still combines partitions and drops OFFSET rows, which Spark evaluates before
+   * dropping.
+   *
+   * Only handle an immediately adjacent native projection. A filter or join between the limit and
+   * projection can require more input rows, and a blocking operator must consume its input.
+   * Non-collect LocalLimit/GlobalLimit plans can still evaluate a projection before limiting its
+   * output. That preexisting gap and throwing filter predicates are tracked separately in
+   * https://github.com/apache/datafusion-comet/issues/7147.
+   */
+  private def limitCollectProjectionInput(op: SparkPlan): SparkPlan = op match {
+    case limit: CollectLimitExec if limit.limit >= 0 =>
+      limit.child match {
+        case project: CometProjectExec =>
+          // Selecting or renaming columns cannot throw. Keep Project -> Filter adjacent so
+          // native planning can prune columns used only by the filter's predicate.
+          val onlyColumns = project.projectList.forall {
+            case _: Attribute | Alias(_: Attribute, _) => true
+            case _ => false
+          }
+          val alreadyCapped = project.child match {
+            case cap: CometLocalLimitExec => cap.limit <= limit.limit
+            case _ => false
+          }
+          if (onlyColumns || alreadyCapped) {
+            op

Review Comment:
   Confirmed equivalent to the two early `return`s it replaces: `alreadyCapped` is true only for a `CometLocalLimitExec` child with `cap.limit <= limit.limit` and false for any other child, which is exactly when the old code returned `op`. The wider-existing-cap case (inner `LIMIT 10` under an outer collect `LIMIT 2`) still falls through and gets the tighter cap.
   
   Reads better than what I was commenting on. No further changes from me.



##########
docs/source/user-guide/latest/compatibility/index.md:
##########
@@ -106,6 +106,32 @@ Separately, Comet's native Parquet scan currently rejects string columns whose s
 valid UTF-8 rather than reading them like Spark
 ([#4121](https://github.com/apache/datafusion-comet/issues/4121)).
 
+## ANSI errors from rows skipped by LIMIT
+
+Native operators evaluate batches of rows. With `spark.sql.ansi.enabled=true`, some `LIMIT`
+plan shapes can therefore raise an expression error from a row that Spark stops before evaluating.
+For example, with a Parquet batch read in the order `(id, value) = (0, '0'), (1, '1')`:
+
+```sql
+SELECT value FROM t
+WHERE CAST(IF(id = 1, 'bad', value) AS INT) >= 0
+LIMIT 1
+```
+
+Spark returns `'0'` after the first row matches. Comet can evaluate the filter predicate on the
+second row in the same batch and raise `CAST_INVALID_INPUT`. SQL without `ORDER BY` does not
+guarantee this input order; the example describes a physical scan order that exposes the difference.
+
+Comet caps projection input when a collect limit directly consumes the projection. This protects
+that projection's expressions, but does not bound evaluation inside its child operators. Throwing
+filter predicates and projections beneath non-collect `LocalLimit` / `GlobalLimit` plans, such as
+a `LIMIT` inside a subquery, remain affected. These existing limitations and the investigation of
+already-ordered `TakeOrderedAndProject` inputs are tracked in
+[#7147](https://github.com/apache/datafusion-comet/issues/7147).
+
+An error on a row absent from the final result is not always a mismatch: Spark can also need to
+evaluate that row, for example to sort the input or process an `OFFSET` prefix.

Review Comment:
   This closing paragraph is the most useful sentence in the section. "An error on a row absent from the final result is not always a mismatch" is exactly the distinction that makes the rest actionable — without it, a user hitting an ANSI error under a `LIMIT` has no way to tell whether they found a Comet bug or correct Spark behavior they should expect, and would reasonably file either way.
   
   Same for noting that SQL without `ORDER BY` does not guarantee the input order the example relies on. That keeps the example honest about being a physical-order illustration rather than a contract.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: cap native projection input before collect limits [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #6035:
URL: https://github.com/apache/datafusion-comet/pull/6035#discussion_r4321727663


##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -724,6 +724,54 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
     }
   }
 
+  /**
+   * Spark's collect limit consumes at most `limit` rows per partition, but a native projection
+   * evaluates its whole input batch before that limit can stop reading. Limit the projection's
+   * input first so an expression on an unconsumed row cannot raise an error. Keep the outer
+   * limit: it still combines partitions and drops OFFSET rows, which Spark evaluates before
+   * dropping.
+   *
+   * Only handle an immediately adjacent native projection. A filter or join between the limit and
+   * projection can require more input rows, and a blocking operator must consume its input.
+   */

Review Comment:
   This is the part I would most like your read on.
   
   `CollectLimitExec` is not the only operator that stops consuming before its child finishes a batch. `BaseLimitExec.doExecute` is just as lazy:
   
   ```scala
   protected override def doExecute(): RDD[InternalRow] = child.execute().mapPartitionsInternal {
     iter => iter.take(limit)
   }
   ```
   
   and `LocalLimitExec` / `GlobalLimitExec` both have their own Comet serdes (`classOf[LocalLimitExec] -> CometLocalLimitExec`, `classOf[GlobalLimitExec] -> CometGlobalLimitExec`), so the shape is reachable. Spark plans a non-collect limit as `GlobalLimitExec(limit, LocalLimitExec(limit, child), offset)` (`SparkStrategies.scala:1213`), which puts a `LocalLimitExec -> ProjectExec` edge in plans like `SELECT CAST(IF(...) AS INT) FROM t LIMIT 1` used as a subquery or insert source rather than a driver collect. `CollectTailExec` and `TakeOrderedAndProjectExec` are in the same family.
   
   The scaladoc says "Only handle an immediately adjacent native projection", which explains the filter/join exclusion well, but the commit message's "Other plan shapes are outside this PR" is the part that needs a reason rather than a boundary. Is a `LocalLimit -> CometProject` edge safe for a reason I am missing — because the native limit is itself pushed into the native plan and bounds the scan upstream, say — or does it have the same defect?
   
   If it is safe, that reason belongs here in the scaladoc, so the next person does not have to re-derive it. If it is not, it is the same bug under a different operator and I would rather see an issue tracking it than silence, even if the fix stays out of this PR.



##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -724,6 +724,54 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
     }
   }
 
+  /**
+   * Spark's collect limit consumes at most `limit` rows per partition, but a native projection
+   * evaluates its whole input batch before that limit can stop reading. Limit the projection's
+   * input first so an expression on an unconsumed row cannot raise an error. Keep the outer
+   * limit: it still combines partitions and drops OFFSET rows, which Spark evaluates before
+   * dropping.
+   *
+   * Only handle an immediately adjacent native projection. A filter or join between the limit and
+   * projection can require more input rows, and a blocking operator must consume its input.
+   */
+  private def limitCollectProjectionInput(op: SparkPlan): SparkPlan = op match {
+    case limit: CollectLimitExec if limit.limit >= 0 =>
+      limit.child match {
+        case project: CometProjectExec =>
+          // Selecting or renaming columns cannot throw. Keep Project -> Filter adjacent so
+          // native planning can prune columns used only by the filter's predicate.
+          val onlyColumns = project.projectList.forall {
+            case _: Attribute | Alias(_: Attribute, _) => true
+            case _ => false
+          }
+          if (onlyColumns) return op

Review Comment:
   Minor style point: the two early `return`s inside the `case project: CometProjectExec` block make the control flow harder to follow than the `for`-comprehension below, especially since `return` inside a nested pattern match in Scala is easy to misread as returning from the `match` rather than the method.
   
   A guard on the case, or folding both conditions into one `if (onlyColumns || alreadyCapped) op else { ... }`, would read more directly. Not worth a reroll on its own.



##########
spark/src/test/scala/org/apache/comet/CometNativeCastSuite.scala:
##########
@@ -65,6 +66,112 @@ class CometNativeCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
 
   import testImplicits._
 
+  test("LIMIT skips a failing conditional literal cast and retains Comet projection") {
+    withTempPath { path =>
+      withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+        spark
+          .range(5)
+          .selectExpr("id", "CAST(id AS STRING) AS value")
+          .coalesce(1)
+          .write
+          .parquet(path.getCanonicalPath)
+      }
+      withSQLConf(
+        SQLConf.ANSI_ENABLED.key -> "true",
+        CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key -> "true",
+        CometConf.COMET_BATCH_SIZE.key -> "8192") {
+        withParquetTable(path.getCanonicalPath, "cast_branch_rows") {
+          // Both rows share a batch. The native projection must receive only row 0: masking
+          // the cast by its IF condition alone would still evaluate the failing row 1.
+          val cast = "CAST(IF(id = 1, 'bad', value) AS INT)"
+          val masked = s"SELECT $cast AS parsed FROM cast_branch_rows LIMIT 1"
+          // ConstantFolding must leave the throwing literal cast inside the conditional.
+          val badCasts = sql(masked).queryExecution.optimizedPlan
+            .flatMap(_.expressions)
+            .flatMap(_.collect {
+              case c: Cast
+                  if c.child.isInstanceOf[Literal] &&
+                    Option(c.child.asInstanceOf[Literal].value).exists(_.toString == "bad") =>
+                c
+            })
+          assert(badCasts.nonEmpty)
+          val (_, plan) = checkSparkAnswerAndOperator(
+            sql(masked),
+            includeClasses = Seq(classOf[CometProjectExec]))
+          assert(new ExtendedExplainInfo().getCodegenDispatchExpressions(plan).contains("cast"))
+          checkSparkError(
+            sql(s"SELECT $cast FROM cast_branch_rows LIMIT 2"),
+            "CAST_INVALID_INPUT")
+
+          // CollectLimit evaluates the prefix before dropping OFFSET rows. Moving an offset
+          // below the projection would incorrectly hide this error in the skipped first row.
+          checkSparkError(
+            sql(
+              "SELECT CAST(IF(id = 0, 'bad', value) AS INT) " +
+                "FROM cast_branch_rows LIMIT 1 OFFSET 1"),
+            "CAST_INVALID_INPUT")
+
+          withSQLConf(CometConf.COMET_BATCH_SIZE.key -> "2") {

Review Comment:
   This cross-batch case with `COMET_BATCH_SIZE = 2` is the most valuable test in the set — it is what proves the cap is cumulative across batches rather than per batch, which a native `LocalLimit` could plausibly have got wrong. Same for the per-partition test above that reads through `executeColumnar` rather than `executeCollect`, since `executeCollect` can stop after the first partition and would hide a missing cap on the second.
   
   No change requested; flagging that these two are the ones worth keeping if the suite ever gets trimmed for runtime.



##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -724,6 +724,54 @@ case class CometExecRule(session: SparkSession, queryStagePrep: Boolean = false)
     }
   }
 
+  /**
+   * Spark's collect limit consumes at most `limit` rows per partition, but a native projection
+   * evaluates its whole input batch before that limit can stop reading. Limit the projection's
+   * input first so an expression on an unconsumed row cannot raise an error. Keep the outer
+   * limit: it still combines partitions and drops OFFSET rows, which Spark evaluates before
+   * dropping.
+   *
+   * Only handle an immediately adjacent native projection. A filter or join between the limit and
+   * projection can require more input rows, and a blocking operator must consume its input.
+   */
+  private def limitCollectProjectionInput(op: SparkPlan): SparkPlan = op match {
+    case limit: CollectLimitExec if limit.limit >= 0 =>
+      limit.child match {
+        case project: CometProjectExec =>
+          // Selecting or renaming columns cannot throw. Keep Project -> Filter adjacent so
+          // native planning can prune columns used only by the filter's predicate.
+          val onlyColumns = project.projectList.forall {
+            case _: Attribute | Alias(_: Attribute, _) => true
+            case _ => false
+          }
+          if (onlyColumns) return op
+          project.child match {
+            case cap: CometLocalLimitExec if cap.limit <= limit.limit => return op
+            case _ =>
+          }
+          val bounded = for {
+            cap <- convertToComet(LocalLimitExec(limit.limit, project.child), CometLocalLimitExec)

Review Comment:
   `convertToComet` is doing double duty here. Beyond returning the converted plan, on failure it runs `rollUpFallbackReasons(op)` and `reportUnexplainedFallback(op)` against the node passed in — and the node passed in here is a synthetic `LocalLimitExec` that exists only inside this method and is discarded when conversion fails.
   
   I checked and this is currently harmless: every decline path in `isOperatorEnabled` tags a reason, so the strict-mode `IllegalStateException` cannot fire, and the discarded node's tag never reaches EXPLAIN. So this is not a bug today.
   
   It is worth a brief note in the comment, though, that the synthetic node is deliberately routed through `convertToComet` for its config and serde gating, and that its fallback tags are discarded with it. Otherwise the coupling is invisible, and a future change to `reportUnexplainedFallback`'s contract — e.g. making it strict whenever the tag is absent — would fail here for a reason that has nothing to do with the user's plan.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org