You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2024/03/10 21:13:55 UTC

[PR] fix: CometExecRule should handle ShuffleQueryStage and ReusedExchange [arrow-datafusion-comet]

viirya opened a new pull request, #186:
URL: https://github.com/apache/arrow-datafusion-comet/pull/186

   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   Closes #185.
   
   ## Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   ## What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ## How are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: CometExecRule should handle ShuffleQueryStage and ReusedExchange [arrow-datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #186:
URL: https://github.com/apache/arrow-datafusion-comet/pull/186#discussion_r1522275579


##########
spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala:
##########
@@ -222,12 +222,16 @@ class CometSparkSessionExtensions
      */
     // spotless:on
     private def transform(plan: SparkPlan): SparkPlan = {
-      def transform1(op: UnaryExecNode): Option[Operator] = {
-        op.child match {
-          case childNativeOp: CometNativeExec =>
-            QueryPlanSerde.operator2Proto(op, childNativeOp.nativeOp)
-          case _ =>
-            None
+      def transform1(op: SparkPlan): Option[Operator] = {
+        val allNativeExec = op.children.map {

Review Comment:
   They are not UnaryExecNode. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: CometExecRule should handle ShuffleQueryStage and ReusedExchange [arrow-datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #186:
URL: https://github.com/apache/arrow-datafusion-comet/pull/186#discussion_r1519043640


##########
spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala:
##########
@@ -378,6 +382,27 @@ class CometSparkSessionExtensions
             case None => b
           }
 
+        case s @ ShuffleQueryStageExec(_, _: CometShuffleExchangeExec, _) =>

Review Comment:
   Okay



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: CometExecRule should handle ShuffleQueryStage and ReusedExchange [arrow-datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #186:
URL: https://github.com/apache/arrow-datafusion-comet/pull/186#discussion_r1522276365


##########
spark/src/test/scala/org/apache/comet/exec/CometColumnarShuffleSuite.scala:
##########
@@ -933,6 +933,54 @@ class CometShuffleSuite extends CometColumnarShuffleSuite {
   override protected val asyncShuffleEnable: Boolean = false
 
   protected val adaptiveExecutionEnabled: Boolean = true
+
+  import testImplicits._
+
+  test("Comet native operator after ShuffleQueryStage") {
+    withSQLConf(
+      SQLConf.ADAPTIVE_AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+      SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+      CometConf.COMET_EXEC_ENABLED.key -> "true",
+      CometConf.COMET_EXEC_SHUFFLE_ENABLED.key -> "true",
+      CometConf.COMET_COLUMNAR_SHUFFLE_ENABLED.key -> "true") {
+      withParquetTable((0 until 10).map(i => (i, i % 5)), "tbl_a") {
+        withParquetTable((0 until 10).map(i => (i % 10, i + 2)), "tbl_b") {

Review Comment:
   Yea, I copied it from other PR and modified it. I will remove it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: CometExecRule should handle ShuffleQueryStage and ReusedExchange [arrow-datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #186:
URL: https://github.com/apache/arrow-datafusion-comet/pull/186#issuecomment-1995248894

   Merged. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: CometExecRule should handle ShuffleQueryStage and ReusedExchange [arrow-datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya merged PR #186:
URL: https://github.com/apache/arrow-datafusion-comet/pull/186


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: CometExecRule should handle ShuffleQueryStage and ReusedExchange [arrow-datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #186:
URL: https://github.com/apache/arrow-datafusion-comet/pull/186#discussion_r1522628382


##########
spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala:
##########
@@ -222,12 +222,16 @@ class CometSparkSessionExtensions
      */
     // spotless:on
     private def transform(plan: SparkPlan): SparkPlan = {
-      def transform1(op: UnaryExecNode): Option[Operator] = {
-        op.child match {
-          case childNativeOp: CometNativeExec =>
-            QueryPlanSerde.operator2Proto(op, childNativeOp.nativeOp)
-          case _ =>
-            None
+      def transform1(op: SparkPlan): Option[Operator] = {

Review Comment:
   curious that whether this change is related to this PR



##########
spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala:
##########
@@ -378,6 +382,31 @@ class CometSparkSessionExtensions
             case None => b
           }
 
+        // For AQE shuffle stage on a Comet shuffle exchange
+        case s @ ShuffleQueryStageExec(_, _: CometShuffleExchangeExec, _) =>
+          val newOp = transform1(s)
+          newOp match {
+            case Some(nativeOp) =>
+              CometSinkPlaceHolder(nativeOp, s, s)
+            case None =>
+              s
+          }
+
+        // For AQE shuffle stage on a reused Comet shuffle exchange
+        // Note that we don't need to handle `ReusedExchangeExec` for non-AQE case, because
+        // the query plan won't be re-optimized/planned in non-AQE mode.
+        case s @ ShuffleQueryStageExec(

Review Comment:
   curious if we can replace this rule with:
   ```scala
           case ReusedExchangeExec(_, op) => op
   ```
   ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: CometExecRule should handle ShuffleQueryStage and ReusedExchange [arrow-datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #186:
URL: https://github.com/apache/arrow-datafusion-comet/pull/186#discussion_r1519004155


##########
spark/src/test/scala/org/apache/comet/exec/CometNativeShuffleSuite.scala:
##########
@@ -64,8 +64,9 @@ class CometNativeShuffleSuite extends CometTestBase with AdaptiveSparkPlanHelper
           val path = new Path(dir.toURI.toString, "test.parquet")
           makeParquetFileAllTypes(path, dictionaryEnabled = dictionaryEnabled, 1000)
           var allTypes: Seq[Int] = (1 to 20)
-          if (isSpark34Plus) {
-            allTypes = allTypes.filterNot(Set(14, 17).contains)
+          if (!isSpark34Plus) {
+            // TODO: Remove this once after https://github.com/apache/arrow/issues/40038 is fixed
+            allTypes = allTypes.filterNot(Set(14).contains)
           }

Review Comment:
   Previously these columns fail on Spark 3.4. I just ran test with them and looks okay now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: CometExecRule should handle ShuffleQueryStage and ReusedExchange [arrow-datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #186:
URL: https://github.com/apache/arrow-datafusion-comet/pull/186#discussion_r1522717865


##########
spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala:
##########
@@ -222,12 +222,16 @@ class CometSparkSessionExtensions
      */
     // spotless:on
     private def transform(plan: SparkPlan): SparkPlan = {
-      def transform1(op: UnaryExecNode): Option[Operator] = {
-        op.child match {
-          case childNativeOp: CometNativeExec =>
-            QueryPlanSerde.operator2Proto(op, childNativeOp.nativeOp)
-          case _ =>
-            None
+      def transform1(op: SparkPlan): Option[Operator] = {

Review Comment:
   These operators are not UnaryExecNode.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: CometExecRule should handle ShuffleQueryStage and ReusedExchange [arrow-datafusion-comet]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #186:
URL: https://github.com/apache/arrow-datafusion-comet/pull/186#issuecomment-1987377329

   ## [Codecov](https://app.codecov.io/gh/apache/arrow-datafusion-comet/pull/186?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `54.16667%` with `11 lines` in your changes are missing coverage. Please review.
   > Project coverage is 33.30%. Comparing base [(`488c523`)](https://app.codecov.io/gh/apache/arrow-datafusion-comet/commit/488c52365fa331734785ec675f02b6a3c4605587?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`30ded81`)](https://app.codecov.io/gh/apache/arrow-datafusion-comet/pull/186?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   
   | [Files](https://app.codecov.io/gh/apache/arrow-datafusion-comet/pull/186?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...org/apache/comet/CometSparkSessionExtensions.scala](https://app.codecov.io/gh/apache/arrow-datafusion-comet/pull/186?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9Db21ldFNwYXJrU2Vzc2lvbkV4dGVuc2lvbnMuc2NhbGE=) | 54.54% | [5 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/arrow-datafusion-comet/pull/186?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [.../scala/org/apache/comet/serde/QueryPlanSerde.scala](https://app.codecov.io/gh/apache/arrow-datafusion-comet/pull/186?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9zZXJkZS9RdWVyeVBsYW5TZXJkZS5zY2FsYQ==) | 50.00% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/arrow-datafusion-comet/pull/186?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##               main     #186      +/-   ##
   ============================================
   + Coverage     33.29%   33.30%   +0.01%     
   - Complexity      766      768       +2     
   ============================================
     Files           107      107              
     Lines         35385    35405      +20     
     Branches       7658     7669      +11     
   ============================================
   + Hits          11781    11792      +11     
   - Misses        21157    21159       +2     
   - Partials       2447     2454       +7     
   ```
   
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/arrow-datafusion-comet/pull/186?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: CometExecRule should handle ShuffleQueryStage and ReusedExchange [arrow-datafusion-comet]

Posted by "snmvaughan (via GitHub)" <gi...@apache.org>.
snmvaughan commented on code in PR #186:
URL: https://github.com/apache/arrow-datafusion-comet/pull/186#discussion_r1519036311


##########
spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala:
##########
@@ -378,6 +382,27 @@ class CometSparkSessionExtensions
             case None => b
           }
 
+        case s @ ShuffleQueryStageExec(_, _: CometShuffleExchangeExec, _) =>

Review Comment:
   I'd suggest highlighting the difference between these 2 blocks with comments.  The actions are exactly the same, so the only difference is the case check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: CometExecRule should handle ShuffleQueryStage and ReusedExchange [arrow-datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #186:
URL: https://github.com/apache/arrow-datafusion-comet/pull/186#discussion_r1522245286


##########
spark/src/test/scala/org/apache/comet/exec/CometColumnarShuffleSuite.scala:
##########
@@ -933,6 +933,54 @@ class CometShuffleSuite extends CometColumnarShuffleSuite {
   override protected val asyncShuffleEnable: Boolean = false
 
   protected val adaptiveExecutionEnabled: Boolean = true
+
+  import testImplicits._
+
+  test("Comet native operator after ShuffleQueryStage") {
+    withSQLConf(
+      SQLConf.ADAPTIVE_AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+      SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+      CometConf.COMET_EXEC_ENABLED.key -> "true",
+      CometConf.COMET_EXEC_SHUFFLE_ENABLED.key -> "true",
+      CometConf.COMET_COLUMNAR_SHUFFLE_ENABLED.key -> "true") {
+      withParquetTable((0 until 10).map(i => (i, i % 5)), "tbl_a") {
+        withParquetTable((0 until 10).map(i => (i % 10, i + 2)), "tbl_b") {

Review Comment:
   I think this is not needed in this test



##########
spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala:
##########
@@ -222,12 +222,16 @@ class CometSparkSessionExtensions
      */
     // spotless:on
     private def transform(plan: SparkPlan): SparkPlan = {
-      def transform1(op: UnaryExecNode): Option[Operator] = {
-        op.child match {
-          case childNativeOp: CometNativeExec =>
-            QueryPlanSerde.operator2Proto(op, childNativeOp.nativeOp)
-          case _ =>
-            None
+      def transform1(op: SparkPlan): Option[Operator] = {
+        val allNativeExec = op.children.map {

Review Comment:
   I don't see why this change is needed here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: CometExecRule should handle ShuffleQueryStage and ReusedExchange [arrow-datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #186:
URL: https://github.com/apache/arrow-datafusion-comet/pull/186#discussion_r1519004412


##########
spark/src/test/scala/org/apache/comet/exec/CometNativeShuffleSuite.scala:
##########
@@ -64,8 +64,9 @@ class CometNativeShuffleSuite extends CometTestBase with AdaptiveSparkPlanHelper
           val path = new Path(dir.toURI.toString, "test.parquet")
           makeParquetFileAllTypes(path, dictionaryEnabled = dictionaryEnabled, 1000)
           var allTypes: Seq[Int] = (1 to 20)
-          if (isSpark34Plus) {
-            allTypes = allTypes.filterNot(Set(14, 17).contains)
+          if (!isSpark34Plus) {
+            // TODO: Remove this once after https://github.com/apache/arrow/issues/40038 is fixed
+            allTypes = allTypes.filterNot(Set(14).contains)
           }

Review Comment:
   Besides, after this change, Comet native operator can be after `CometExchange`. So it triggers the known bug in Java Arrow on column `_14`. I exclude the column for Spark 3.2 and 3.3. 
   
   ```
   == Physical Plan ==                                                                                                                                                                                                                    
   AdaptiveSparkPlan isFinalPlan=true                     
   +- == Final Plan ==                                                                                                
      *(1) ColumnarToRow                                                                                              
      +- CometProject [_1#556], [_1#556]                                                                              
         +- ShuffleQueryStage 0                                                                                                                                                                                                           
            +- CometExchange hashpartitioning(_13#568, 10), REPARTITION_BY_NUM, CometNativeShuffle, [plan_id=838]                                                                                                                         
               +- CometScan parquet [_1#556,_13#568] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/Users/liangchi/repos/arrow-datafusion-comet/spark/target/tmp/spa..., PartitionFilter
   s: [], PushedFilters: [], ReadSchema: struct<_1:boolean,_13:string>                                       
   +- == Initial Plan ==                                                                                              
      CometProject [_1#556], [_1#556]                                                                                 
      +- CometExchange hashpartitioning(_13#568, 10), REPARTITION_BY_NUM, CometNativeShuffle, [plan_id=830]
         +- CometScan parquet [_1#556,_13#568] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/Users/liangchi/repos/arrow-datafusion-comet/spark/target/tmp/spa..., PartitionFilters: [],
    PushedFilters: [], ReadSchema: struct<_1:boolean,_13:string>
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: CometExecRule should handle ShuffleQueryStage and ReusedExchange [arrow-datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #186:
URL: https://github.com/apache/arrow-datafusion-comet/pull/186#discussion_r1522720116


##########
spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala:
##########
@@ -378,6 +382,31 @@ class CometSparkSessionExtensions
             case None => b
           }
 
+        // For AQE shuffle stage on a Comet shuffle exchange
+        case s @ ShuffleQueryStageExec(_, _: CometShuffleExchangeExec, _) =>
+          val newOp = transform1(s)
+          newOp match {
+            case Some(nativeOp) =>
+              CometSinkPlaceHolder(nativeOp, s, s)
+            case None =>
+              s
+          }
+
+        // For AQE shuffle stage on a reused Comet shuffle exchange
+        // Note that we don't need to handle `ReusedExchangeExec` for non-AQE case, because
+        // the query plan won't be re-optimized/planned in non-AQE mode.
+        case s @ ShuffleQueryStageExec(

Review Comment:
   Hmm, then users won't see `ReusedExchangeExec` anymore in explain string or Spark UI. It is useful to know which part is reused by Spark.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: CometExecRule should handle ShuffleQueryStage and ReusedExchange [arrow-datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #186:
URL: https://github.com/apache/arrow-datafusion-comet/pull/186#discussion_r1523625193


##########
spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala:
##########
@@ -378,6 +382,31 @@ class CometSparkSessionExtensions
             case None => b
           }
 
+        // For AQE shuffle stage on a Comet shuffle exchange
+        case s @ ShuffleQueryStageExec(_, _: CometShuffleExchangeExec, _) =>
+          val newOp = transform1(s)
+          newOp match {
+            case Some(nativeOp) =>
+              CometSinkPlaceHolder(nativeOp, s, s)
+            case None =>
+              s
+          }
+
+        // For AQE shuffle stage on a reused Comet shuffle exchange
+        // Note that we don't need to handle `ReusedExchangeExec` for non-AQE case, because
+        // the query plan won't be re-optimized/planned in non-AQE mode.
+        case s @ ShuffleQueryStageExec(

Review Comment:
   oops you are right



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org