[I] Spark 3.4 SQL test SPARK-34637 fails since #6547 because AQE re-plans flip the DPP join's build side [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new issue, #6645:
URL: https://github.com/apache/datafusion-comet/issues/6645

   ### Describe the bug
   
   Spark's `DynamicPartitionPruningV1SuiteAEOn` test "SPARK-34637: DPP side broadcast query stage is created firstly" fails in the Spark 3.4 `sql_core-1` shard:
   
   ```
   hasReuse was false SubqueryBroadcast dynamicpruning#66615, 0, [store_id#66614], [id=#157769]
   ...
   should have been reused in
   AdaptiveSparkPlan isFinalPlan=true
   +- == Final Plan ==
      *(4) BroadcastHashJoin [store_id#65490], [store_id#66614], Inner, BuildLeft, false
   ```
   
   The initial plan broadcasts the right side, which the DPP subquery reuses. The final plan broadcasts the left side instead, so the subquery's broadcast is never reused.
   
   It fails on #4565 (run 37212552551) and on #5841 (run 37222066341). #5841 changes only CI, and both runs merged with `main` after #6547. It passed on #6415's run on 2026-09-29 (`main` at e3d52feaf6). Spark 3.4 SQL tests run only on pull requests that carry `run-spark-3.4-tests`, so neither the merge queue nor the nightly catches it.
   
   ### Cause
   
   #6547 patches the stale `Scan` leaf of a native operator that AQE reuses over a shuffle stage (the final aggregate of a two-phase aggregate). It does this in `CometExecRule` while that rule runs as AQE's query stage preparation rule, so during each re-plan. The re-planned plan then differs from the current one, and AQE adopts it. After that, AQE cannot link the broadcast stage it later builds over the reused operator back into its logical plan, because that operator's logical node now sits inside a `LogicalQueryStage`. The join's build side is no longer pinned.
   
   Once the probe side's shuffle stage materializes, the next re-plan picks the smaller side using runtime statistics. On Spark 3.4 the AQE DPP probe side stays in Spark, and its shuffle reports a smaller size than Comet's, so the join flips to `BuildLeft`.
   
   ### Steps to reproduce
   
   Run the test against Spark 3.4 with Comet enabled. It passes with `spark.comet.shuffle.directRead.enabled=false`, which keeps #6547's refresh from firing, and fails with the default.
   
   ### Expected behavior
   
   AQE re-plans should not change because of Comet's shuffle read refresh, and the DPP broadcast should be reused as it was before #6547.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Spark 3.4 SQL test SPARK-34637 fails since #6547 because AQE re-plans flip the DPP join's build side [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove closed issue #6645: Spark 3.4 SQL test SPARK-34637 fails since #6547 because AQE re-plans flip the DPP join's build side
URL: https://github.com/apache/datafusion-comet/issues/6645


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org