You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2026/07/07 23:28:59 UTC

[PR] [DO-NOT-MERGE][SPARK-58018][SQL][TESTS] Deflake AQE 'should avoid to submit shuffle job on cancellation' test [spark]

HyukjinKwon opened a new pull request, #57096:
URL: https://github.com/apache/spark/pull/57096

   > [!NOTE]
   > Draft/verification PR from a personal fork. The `[TEMP][DO-NOT-MERGE]` fork-only workflow commit (`.github/workflows/zz_fork_aqe_verify.yml`) will be dropped before this is finalized for review.
   
   ### What changes were proposed in this pull request?
   
   Test-only change to deflake `AdaptiveQueryExecSuite`'s `SPARK-47148: AQE should avoid to submit shuffle job on cancellation`.
   
   The test uses a `slow_udf` in scalar subqueries on the `df`/`df3` shuffle stages to delay their submission, so that the `df2` coalesce stage (no subquery) submits its shuffle job and fails with "coalesce test error" first, while the two delayed stages are still sleeping and thus get cancelled before submission. The test then asserts the failure chain contains "coalesce test error" and that exactly the middle stage was materialized.
   
   With a 3000ms sleep this ordering is timing-dependent: under a loaded CI runner the coalesce stage's map task can be slow enough that a delayed stage is submitted, or the coalesce stage is itself cancelled, before "coalesce test error" is thrown — making the error disappear from the failure chain and failing the assertion `errMsgList.exists(_.contains("coalesce test error"))`.
   
   Increase the sleep to 15000ms so the coalesce stage reliably wins the race. The per-test timeout is 20 minutes, so this adds no meaningful cost.
   
   ### Why are the changes needed?
   
   The test is flaky on the `build_java21` lane (observed on branch-4.0), failing with `...contains("coalesce test error") was false`.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No. Test-only.
   
   ### How was this patch tested?
   
   Ran the SPARK-47148 AQE cancellation test repeatedly on GitHub Actions on a fork.
   
   **CI results**
   
   - Failing (before): https://github.com/apache/spark/actions/runs/28778790780
   - Passing (after): <!-- PASS_LINK -->
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Opus 4.8)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58018][SQL][TESTS] Deflake AQE 'should avoid to submit shuffle job on cancellation' test [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #57096:
URL: https://github.com/apache/spark/pull/57096#issuecomment-4910125002

   **Merge Summary:**
   - merged into master https://github.com/apache/spark/commit/d91cf3124375f46dfc87bf451b013a79b360e2ec
   - merged into branch-4.x https://github.com/apache/spark/commit/a550c35d360a09ed3a40818f2780e710ebce8bb8
   - merged into branch-4.2 https://github.com/apache/spark/commit/f29f5cffdcdbd63dcb19a58747384766a6bd5821
   - merged into branch-4.1 https://github.com/apache/spark/commit/982de1377e72ccca671fb92ddba8c64b5ab1d3a9
   - merged into branch-4.0 https://github.com/apache/spark/commit/b240ed2766c1a4dd56db659777ffb32e9a2bcf7b
   
   *Posted by `merge_spark_pr.py`*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58018][SQL][TESTS] Deflake AQE 'should avoid to submit shuffle job on cancellation' test [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #57096: [SPARK-58018][SQL][TESTS] Deflake AQE 'should avoid to submit shuffle job on cancellation' test
URL: https://github.com/apache/spark/pull/57096


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org