You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/10/10 13:27:13 UTC

[PR] fix: preserve columnar output when reverting a cached AQE result stage [datafusion-comet]

andygrove opened a new pull request, #6852:
URL: https://github.com/apache/datafusion-comet/pull/6852

   ## Which issue does this PR close?
   
   Closes #6687.
   
   ## Rationale for this change
   
   `RevertNativeForTransitionHeavyStages` assumed a result stage is always collected as rows and passed `outputColumnar = false` for it. That does not hold for a cached query on Spark 4.0+. When the cache serializer accepts columnar input, as Comet's `ArrowCachedBatchSerializer` does, Spark marks the cached `AdaptiveSparkPlanExec` columnar, AQE plans the final stage with `outputsColumnar = true`, and the cache builder calls `executeColumnar()` on it. Once the rule reverted that stage to a row-based root such as `FilterExec`, nothing restored the columnar output, and building the cache failed with `FilterExec has column support mismatch`.
   
   Transition reversion is off by default, so this only affects users who enable `spark.comet.exec.transitionRevert.enabled`.
   
   ## What changes are included in this PR?
   
   - Both result-stage call sites go through a new `revertResultStageIfNeeded`, which reads the requested format off the stage root with `outputColumnar = plan.supportsColumnar && !plan.supportsRowBased`, as proposed in the issue. Spark's transition insertion leaves a columnar-only root only when columnar output was requested; when rows are wanted it puts a columnar-to-row transition on top. When the format is columnar, the existing logic in `revertStageIfNeeded` wraps a row-based reverted root in `RowToColumnarExec`, which is what Spark itself inserts in that case.
   - Without AQE, Spark always plans for row output, so that path behaves as before.
   
   ## How are these changes tested?
   
   A new test in `RevertNativeForTransitionHeavyStagesSuite` reproduces the issue. With AQE on, `transitionRevert.enabled=true`, `maxTransitions=0` and `spark.comet.exec.filter.enabled=false`, it caches `SELECT _2, s FROM (SELECT _2, sum(_1) AS s FROM tbl GROUP BY _2) WHERE s > 10` and checks the answer against Spark's. It then checks that the cache was built from columnar output, that the result stage was reverted, and that the reverted stage is still columnar. On `main` the test fails with `FilterExec has column support mismatch`.
   
   So that the cache is built from columnar output, the suite now installs `ArrowCachedBatchSerializer` the way `CometInMemoryCacheSuite` does, clearing Spark's memoized serializer before and after the suite. The test runs on Spark 4.0+ and is skipped on 3.x, where AQE builds a cache through rows.
   
   `RevertNativeForTransitionHeavyStagesSuite` passes locally on Spark 3.4, 3.5, 4.0, 4.1 and 4.2. On Spark 4.1, `CometShuffleInputConversionSuite`, `CometExecRuleSuite`, `CometCelebornShufflePlanningSuite` and `CometParquetWriterSuite`, which also exercise the rule, pass as well.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve columnar output when reverting a cached AQE result stage [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #6852:
URL: https://github.com/apache/datafusion-comet/pull/6852#discussion_r4402241920


##########
spark/src/test/scala/org/apache/comet/rules/RevertNativeForTransitionHeavyStagesSuite.scala:
##########
@@ -803,6 +842,104 @@ class RevertNativeForTransitionHeavyStagesSuite extends CometTestBase {
     }
   }
 
+  // https://github.com/apache/datafusion-comet/issues/6687
+  test("transition-heavy revert restores columnar output for a cached AQE result stage") {

Review Comment:
   You are right, the branch does not depend on the Spark version, so the 3.4 and 3.5 profiles should run it. 5fc0358206 adds a rule-level test, "keeps the output format of the result root", that applies the rule to `CometSparkToColumnarExec(CometNativeColumnarToRowExec(cometFilter(leaf)))` over a row-only leaf and asserts a `RowToColumnarExec` over the Spark `FilterExec` with no Comet execs left. It then wraps that root in a `ColumnarToRowExec` and asserts the reverted stage still outputs rows, with no `RowToColumnarExec`. Both run with AQE on and off. I ran the suite on Spark 4.1 and 3.5, and with the branch changed to `outputColumnar = false` both new tests fail on 3.5.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve columnar output when reverting a cached AQE result stage [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #6852:
URL: https://github.com/apache/datafusion-comet/pull/6852#discussion_r4238845482


##########
spark/src/main/scala/org/apache/comet/rules/RevertNativeForTransitionHeavyStages.scala:
##########
@@ -65,19 +65,27 @@ case class RevertNativeForTransitionHeavyStages(session: SparkSession, wholePlan
       case exchange: ShuffleExchangeLike =>
         revertShuffleStageIfNeeded(exchange).getOrElse(plan)
       case _ =>
-        // Result stage: its output is collected as rows.
-        revertStageIfNeeded(plan, outputColumnar = false).getOrElse(plan)
+        revertResultStageIfNeeded(plan).getOrElse(plan)
     }
   }
 
   private def applyForNonAQE(plan: SparkPlan): SparkPlan = {
     val withRevertedStages = plan.transformUp { case exchange: ShuffleExchangeLike =>
       revertShuffleStageIfNeeded(exchange).getOrElse(exchange)
     }
-    revertStageIfNeeded(withRevertedStages, outputColumnar = false)
-      .getOrElse(withRevertedStages)
+    revertResultStageIfNeeded(withRevertedStages).getOrElse(withRevertedStages)
   }
 
+  /**
+   * Reverts the result stage if needed, keeping the output format its consumer reads. That is
+   * usually rows, but on Spark 4.0+, when the cache serializer accepts columnar input, as Comet's
+   * does, Spark marks a cached `AdaptiveSparkPlanExec` columnar and AQE plans its final stage to
+   * output columnar batches. Only then does Spark's transition insertion leave a columnar-only
+   * root; when rows are wanted it puts a columnar-to-row transition on top.
+   */
+  private def revertResultStageIfNeeded(plan: SparkPlan): Option[SparkPlan] =
+    revertStageIfNeeded(plan, outputColumnar = plan.supportsColumnar && !plan.supportsRowBased)

Review Comment:
   What should happen when the root supports both formats? With `supportsColumnar && supportsRowBased` this infers rows. But when columnar output was requested, Spark's `ensureOutputsColumnar` keeps such a root as is, so the root alone doesn't say which format the consumer wants.
   
   Up to 4.1 I couldn't find a way to reach this. The only dual-format operators there are `UnionExec` and `InMemoryTableScanExec`, and neither can have a C2R below it in the same stage. On Spark 4.2, though, `ArrowEvalPythonExec` reports `supportsRowBased = true` and `supportsColumnar = child.supportsColumnar && conf.arrowPySparkUDFColumnarInputEnabled`, and that conf defaults to true. Take a cached query whose result stage is rooted at a pandas UDF over a columnar child with C2Rs below it, for example with `spark.comet.sparkToColumnar.enabled`. The reverted `ArrowEvalPythonExec` becomes row-only, and the cache builder's `executeColumnar()` would hit the same mismatch this PR fixes.
   
   Would it make sense to skip the revert when the original root supports both formats and the reverted root no longer supports columnar? The doc comment above could also note that a dual-format root leaves the requested format unknown.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve columnar output when reverting a cached AQE result stage [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #6852:
URL: https://github.com/apache/datafusion-comet/pull/6852#discussion_r4265127983


##########
spark/src/main/scala/org/apache/comet/rules/RevertNativeForTransitionHeavyStages.scala:
##########
@@ -65,19 +65,39 @@ case class RevertNativeForTransitionHeavyStages(session: SparkSession, wholePlan
       case exchange: ShuffleExchangeLike =>
         revertShuffleStageIfNeeded(exchange).getOrElse(plan)
       case _ =>
-        // Result stage: its output is collected as rows.
-        revertStageIfNeeded(plan, outputColumnar = false).getOrElse(plan)
+        revertResultStageIfNeeded(plan).getOrElse(plan)
     }
   }
 
   private def applyForNonAQE(plan: SparkPlan): SparkPlan = {
     val withRevertedStages = plan.transformUp { case exchange: ShuffleExchangeLike =>
       revertShuffleStageIfNeeded(exchange).getOrElse(exchange)
     }
-    revertStageIfNeeded(withRevertedStages, outputColumnar = false)
-      .getOrElse(withRevertedStages)
+    revertResultStageIfNeeded(withRevertedStages).getOrElse(withRevertedStages)
   }
 
+  /**
+   * Reverts the result stage if needed, keeping the output format its consumer reads. That is
+   * usually rows, but on Spark 4.0+, when the cache serializer accepts columnar input, as Comet's
+   * does, Spark marks a cached `AdaptiveSparkPlanExec` columnar and AQE plans its final stage to
+   * output columnar batches. Only then does Spark's transition insertion leave a columnar-only
+   * root; when rows are wanted it puts a columnar-to-row transition on top.
+   *
+   * Spark leaves a root that supports both formats as it is in either case, so such a root, for
+   * example `ArrowEvalPythonExec` over a columnar child on Spark 4.2, does not say which format
+   * its consumer reads. No transition provides both, so that stage is reverted only if the
+   * reverted root supports both formats too.
+   */
+  private def revertResultStageIfNeeded(plan: SparkPlan): Option[SparkPlan] =
+    if (plan.supportsColumnar && plan.supportsRowBased) {

Review Comment:
   You're right. Without AQE, `CachedRDDBuilder` reads `cachedPlan.supportsColumnar` from the plan this rule leaves, and every other consumer reads rows, so the check bought nothing there and only kept the stage native. In bc5bddc208 it applies only when `spark.sql.adaptive.enabled` is set, read from the conf rather than from `wholePlan` as you suggested, and without AQE a root that supports both formats is reverted to row output as on `main`.
   
   The test now runs under AQE with `wholePlan` off and on, where both stages behave as before, and without AQE, where both are reverted to row output. Keying the check on `wholePlan` fails the preview case, and dropping the AQE condition fails the case without AQE. I've updated the description to match.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve columnar output when reverting a cached AQE result stage [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #6852:
URL: https://github.com/apache/datafusion-comet/pull/6852#discussion_r4260562971


##########
spark/src/main/scala/org/apache/comet/rules/RevertNativeForTransitionHeavyStages.scala:
##########
@@ -65,19 +65,27 @@ case class RevertNativeForTransitionHeavyStages(session: SparkSession, wholePlan
       case exchange: ShuffleExchangeLike =>
         revertShuffleStageIfNeeded(exchange).getOrElse(plan)
       case _ =>
-        // Result stage: its output is collected as rows.
-        revertStageIfNeeded(plan, outputColumnar = false).getOrElse(plan)
+        revertResultStageIfNeeded(plan).getOrElse(plan)
     }
   }
 
   private def applyForNonAQE(plan: SparkPlan): SparkPlan = {
     val withRevertedStages = plan.transformUp { case exchange: ShuffleExchangeLike =>
       revertShuffleStageIfNeeded(exchange).getOrElse(exchange)
     }
-    revertStageIfNeeded(withRevertedStages, outputColumnar = false)
-      .getOrElse(withRevertedStages)
+    revertResultStageIfNeeded(withRevertedStages).getOrElse(withRevertedStages)
   }
 
+  /**
+   * Reverts the result stage if needed, keeping the output format its consumer reads. That is
+   * usually rows, but on Spark 4.0+, when the cache serializer accepts columnar input, as Comet's
+   * does, Spark marks a cached `AdaptiveSparkPlanExec` columnar and AQE plans its final stage to
+   * output columnar batches. Only then does Spark's transition insertion leave a columnar-only
+   * root; when rows are wanted it puts a columnar-to-row transition on top.
+   */
+  private def revertResultStageIfNeeded(plan: SparkPlan): Option[SparkPlan] =
+    revertStageIfNeeded(plan, outputColumnar = plan.supportsColumnar && !plan.supportsRowBased)

Review Comment:
   You're right, the root alone can't tell us which format was requested when it supports both. There was a second way to hit it, too. If the reverted root still supported both formats, `revertStageIfNeeded` put a `ColumnarToRowExec` over it, and a columnar consumer failed the same way.
   
   In 7552039264, a root that supports both formats is reverted only when the reverted root supports both as well, and it goes back with no transition on top. Otherwise the stage stays native, since no transition gives both formats. The doc comment now says that such a root leaves the requested format unknown.
   
   PySpark can't run in our JVM suites, so the new test uses a stand-in operator that reports its formats the way `ArrowEvalPythonExec` does on 4.2. It covers both cases with AQE on and off, and it fails against the previous commit.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve columnar output when reverting a cached AQE result stage [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6852:
URL: https://github.com/apache/datafusion-comet/pull/6852#issuecomment-6477747420

   Thanks @viirya!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve columnar output when reverting a cached AQE result stage [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6852:
URL: https://github.com/apache/datafusion-comet/pull/6852#issuecomment-6110922330

   @peterxcli you worked on the in-memory cache in #5869, so you know this area. If you have time, would you be able to review this one? Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve columnar output when reverting a cached AQE result stage [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #6852:
URL: https://github.com/apache/datafusion-comet/pull/6852#discussion_r4379457445


##########
spark/src/test/scala/org/apache/comet/rules/RevertNativeForTransitionHeavyStagesSuite.scala:
##########
@@ -803,6 +842,104 @@ class RevertNativeForTransitionHeavyStagesSuite extends CometTestBase {
     }
   }
 
+  // https://github.com/apache/datafusion-comet/issues/6687
+  test("transition-heavy revert restores columnar output for a cached AQE result stage") {

Review Comment:
   The new `outputColumnar = !plan.supportsRowBased` branch for a columnar-only root is only exercised by this test, which is skipped before Spark 4.0, and the suite has no assertion on `RowToColumnarExec` (the only mentions are in a doc comment). The branch itself is version independent, so the 3.4 and 3.5 profiles never run it. Could a small rule-level case cover it, like the `BothFormatsExec` test already does for the dual-format root? For example, apply the rule to `CometSparkToColumnarExec(CometNativeColumnarToRowExec(cometFilter(leaf)))`, which is columnar-only with one transition, and assert a `RowToColumnarExec` over a reverted root with no Comet execs left, plus the matching row case where a `ColumnarToRowExec` root stays row output. I did not run the suite.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve columnar output when reverting a cached AQE result stage [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #6852:
URL: https://github.com/apache/datafusion-comet/pull/6852#discussion_r4263106035


##########
spark/src/main/scala/org/apache/comet/rules/RevertNativeForTransitionHeavyStages.scala:
##########
@@ -65,19 +65,39 @@ case class RevertNativeForTransitionHeavyStages(session: SparkSession, wholePlan
       case exchange: ShuffleExchangeLike =>
         revertShuffleStageIfNeeded(exchange).getOrElse(plan)
       case _ =>
-        // Result stage: its output is collected as rows.
-        revertStageIfNeeded(plan, outputColumnar = false).getOrElse(plan)
+        revertResultStageIfNeeded(plan).getOrElse(plan)
     }
   }
 
   private def applyForNonAQE(plan: SparkPlan): SparkPlan = {
     val withRevertedStages = plan.transformUp { case exchange: ShuffleExchangeLike =>
       revertShuffleStageIfNeeded(exchange).getOrElse(exchange)
     }
-    revertStageIfNeeded(withRevertedStages, outputColumnar = false)
-      .getOrElse(withRevertedStages)
+    revertResultStageIfNeeded(withRevertedStages).getOrElse(withRevertedStages)
   }
 
+  /**
+   * Reverts the result stage if needed, keeping the output format its consumer reads. That is
+   * usually rows, but on Spark 4.0+, when the cache serializer accepts columnar input, as Comet's
+   * does, Spark marks a cached `AdaptiveSparkPlanExec` columnar and AQE plans its final stage to
+   * output columnar batches. Only then does Spark's transition insertion leave a columnar-only
+   * root; when rows are wanted it puts a columnar-to-row transition on top.
+   *
+   * Spark leaves a root that supports both formats as it is in either case, so such a root, for
+   * example `ArrowEvalPythonExec` over a columnar child on Spark 4.2, does not say which format
+   * its consumer reads. No transition provides both, so that stage is reverted only if the
+   * reverted root supports both formats too.
+   */
+  private def revertResultStageIfNeeded(plan: SparkPlan): Option[SparkPlan] =
+    if (plan.supportsColumnar && plan.supportsRowBased) {

Review Comment:
   Does the non-AQE path need this check? Without AQE, I think the consumer adapts to whatever root the rule leaves. `CachedRDDBuilder` picks `executeColumnar()` or `execute()` from `cachedPlan.supportsColumnar` on the final plan, after this rule has run, and the other consumers read rows. Only AQE fixes `AdaptiveSparkPlanExec.supportsColumnar = true` before it plans the final stage, so that is the only place where a row-only reverted root breaks the consumer.
   
   So a non-AQE dual-format root could still be reverted the way `main` does it. Keeping it native there gives up the revert for no correctness gain. It also differs from "Without AQE, Spark always plans for row output, so that path behaves as before" in the description, and the `AQE=false` case of the new test pins that behavior.
   
   Would it work to apply the dual-format check only when `adaptiveExecutionEnabled` is set? Keying it on the AQE conf rather than on `wholePlan` would keep the plan-only preview under AQE consistent with what AQE executes. The `AQE=false` case of the test could then check that the stage is reverted.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve columnar output when reverting a cached AQE result stage [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6852:
URL: https://github.com/apache/datafusion-comet/pull/6852#issuecomment-6148969178

   @viirya thanks for the review. I've answered the dual-format question inline.
   
   On the backport, I'd keep this on main. On branch-1.1 the crash needs Comet's cache serializer, which `spark.comet.exec.inMemoryCache.enabled` installs, plus `spark.comet.exec.transitionRevert.enabled`, and both are off by default there. It isn't a regression either. 1.0 had no Arrow cache serializer, so with Spark's default one the cache never asked a reverted stage for columnar output. The pick also conflicts in both files, so it would be a manual port.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: preserve columnar output when reverting a cached AQE result stage [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #6852:
URL: https://github.com/apache/datafusion-comet/pull/6852


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org