You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "JkSelf (via GitHub)" <gi...@apache.org> on 2025/08/15 07:59:31 UTC
[PR] [SPARK-53289] Invoke cancelJobGroup before start passing the ExecutionEnd event to prevent execution of canceled tasks. [spark]
JkSelf opened a new pull request, #52039:
URL: https://github.com/apache/spark/pull/52039
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
We optimized Gluten's broadcast hash join by ensuring the hash table is built only once per executor. When the ExecutionEnd event is received, the corresponding hash table is released. However, while running Gluten unit test, we encountered an exception:
After the ExecutionEnd event was triggered and the hash table was released, the hash table was unexpectedly recreated, leading to a core dump. After investigating with @wangyum , we found that the unit test result was None, which triggered the AQEPropagateEmptyRelation rule to skip the join operation. Despite this, the task was not actually canceled, which caused the hash table to be rebuilt after it had already been released.
This PR ensures that the task is properly canceled in the finally block to prevent such issues.
### Why are the changes needed?
Bug fix.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing tests.
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2286777579
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -195,6 +195,12 @@ object SQLExecution extends Logging {
}
}
}
+
+ if (executionId == rootExecutionId) {
Review Comment:
@cloud-fan Yes. Added the comments. Please help to review again. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2284133597
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -195,6 +195,10 @@ object SQLExecution extends Logging {
}
}
}
+
+ sparkSession.sparkContext.cancelJobsWithTag(
+ executionIdJobTag(sparkSession, executionId))
Review Comment:
> nested query execution can also hit this code branch, or did I miss something?
Yes. We need skip this cancel operator for nested query execution. Already add the check only cancel in root execution. Please help to review again. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobGroup before start passing the ExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2281153542
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -110,6 +110,8 @@ object SQLExecution extends Logging {
redactedStr.substring(0, Math.min(truncateLength, redactedStr.length))
}.getOrElse(callSite.shortForm)
+ sparkSession.sparkContext.setJobGroup(executionId.toString, desc, true)
Review Comment:
I don't quite get it. We have landed a fix to cancel broadcasted jobs properly by using tags, and it is fixed in https://github.com/apache/spark/pull/41440. You can't use the job groups cuz it overwrites the existent job groups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2501564217
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLExecutionSuite.scala:
##########
@@ -50,7 +50,7 @@ class SQLExecutionSuite extends SparkFunSuite with SQLConfHelper {
}
}
- test("concurrent query execution with fork-join pool (SPARK-13747)") {
+ ignore("concurrent query execution with fork-join pool (SPARK-13747)") {
Review Comment:
Yes. It is only be cancelled once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #52039:
URL: https://github.com/apache/spark/pull/52039#issuecomment-3421130975
@JkSelf it looks like the CI build failed on this. could you please do a rebase? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2468050710
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLExecutionSuite.scala:
##########
@@ -50,7 +50,7 @@ class SQLExecutionSuite extends SparkFunSuite with SQLConfHelper {
}
}
- test("concurrent query execution with fork-join pool (SPARK-13747)") {
+ ignore("concurrent query execution with fork-join pool (SPARK-13747)") {
Review Comment:
CC @wangyum Could you also help to take a look? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2284746769
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -195,6 +195,12 @@ object SQLExecution extends Logging {
}
}
}
+
+ if (executionId == rootExecutionId) {
Review Comment:
let's leave some code comments here to explain it.
It will be better if we can cancel uesless jobs earlier when nested query executions are completed, but we can do it in a followup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2283934935
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -195,6 +195,10 @@ object SQLExecution extends Logging {
}
}
}
+
+ sparkSession.sparkContext.cancelJobsWithTag(
+ executionIdJobTag(sparkSession, executionId))
Review Comment:
We should only do this if this is the root execution
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -195,6 +195,10 @@ object SQLExecution extends Logging {
}
}
}
+
+ sparkSession.sparkContext.cancelJobsWithTag(
+ executionIdJobTag(sparkSession, executionId))
Review Comment:
BTW does it work for AQE?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobGroup before start passing the ExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2281360580
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -110,6 +110,8 @@ object SQLExecution extends Logging {
redactedStr.substring(0, Math.min(truncateLength, redactedStr.length))
}.getOrElse(callSite.shortForm)
+ sparkSession.sparkContext.setJobGroup(executionId.toString, desc, true)
Review Comment:
@HyukjinKwon Thanks for your explanations. I have remove this change. And directly use `cancelJobsWithTag `API to cancel the tasks. Could you help to review again? Thanks for your help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2468057191
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLExecutionSuite.scala:
##########
@@ -50,7 +50,7 @@ class SQLExecutionSuite extends SparkFunSuite with SQLConfHelper {
}
}
- test("concurrent query execution with fork-join pool (SPARK-13747)") {
+ ignore("concurrent query execution with fork-join pool (SPARK-13747)") {
Review Comment:
The failed unit tests is `SQLExecutionSuite#concurrent query execution with fork-join pool (SPARK-13747)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2469504635
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLExecutionSuite.scala:
##########
@@ -50,7 +50,7 @@ class SQLExecutionSuite extends SparkFunSuite with SQLConfHelper {
}
}
- test("concurrent query execution with fork-join pool (SPARK-13747)") {
+ ignore("concurrent query execution with fork-join pool (SPARK-13747)") {
Review Comment:
The first time is in
`Line 10: 2025-10-27T05:32:55.7668804Z 05:32:55.765 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.0 in stage 29.0 (TID 56) (localhost executor driver): TaskKilled (Stage cancelled: [SPARK_JOB_CANCELLED] Job 20 cancelled part of cancelled job tags spark-session-42c4d99b-c576-4172-b5ed-56c61bb79f14-execution-root-id-25164 SQLSTATE: XXKDA)`
And the second time is in
`Line 73: 2025-10-27T05:32:57.1915923Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[31m org.apache.spark.SparkException: [SPARK_JOB_CANCELLED] Job 20 cancelled part of cancelled job tags spark-session-42c4d99b-c576-4172-b5ed-56c61bb79f14-execution-root-id-25164 SQLSTATE: XXKDA�[0m�[0m`
@cloud-fan Correct me if wrong understanding. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2446793618
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -211,6 +211,20 @@ object SQLExecution extends Logging {
}
}
}
+
+ // Cancel all spark jobs associated with this executionID, but only if it's the
+ // root execution.
+
+ // TODO: Consider enhancing this logic to cancel jobs earlier when nested
+ // query executions are completed.
+ if (executionId == rootExecutionId) {
+ sparkSession.sparkContext.cancelJobsWithTag(
+ executionIdJobTag(sparkSession, executionId))
+ // scalastyle:off println
+ println("cancel the executionID " + executionIdJobTag(sparkSession, executionId))
Review Comment:
we should use `logInfo` instead of print
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobGroup before start passing the ExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on PR #52039:
URL: https://github.com/apache/spark/pull/52039#issuecomment-3199024408
> > This PR ensures that the task is properly canceled ...
>
> How is this possible? Driver can only send an RPC to kill a task, which is not guaranteed to success. The only reliable way to is to shut down the corresponding executors forciably, which is the job of task reaper ([SPARK-18761](https://issues.apache.org/jira/browse/SPARK-18761)).
@cloud-fan Thanks for your review.
I tried removing `sparkSession.sparkContext.cancelJobsWithTag(executionIdJobTag(sparkSession, executionId))` and directly setting `spark.task.reaper.enabled` to `true`, but it doesn't seem to take effect. My understanding is that we still need to call `cancelJobsWithTag `to mark the task as killed, and then enable `spark.task.reaper.enabled` to forcibly kill the corresponding executor to ensure the task is eventually terminated. Is that correct?"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2467624448
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLExecutionSuite.scala:
##########
@@ -50,7 +50,7 @@ class SQLExecutionSuite extends SparkFunSuite with SQLConfHelper {
}
}
- test("concurrent query execution with fork-join pool (SPARK-13747)") {
+ ignore("concurrent query execution with fork-join pool (SPARK-13747)") {
Review Comment:
@cloud-fan Thanks for your reply.
Here is the error stack.
```
org.apache.spark.SparkException: [SPARK_JOB_CANCELLED] Job 20 cancelled part of cancelled job tags spark-session-42c4d99b-c576-4172-b5ed-56c61bb79f14-execution-root-id-25164 SQLSTATE: XXKDA[0m[0m
at org.apache.spark.errors.SparkCoreErrors$.sparkJobCancelled(SparkCoreErrors.scala:222)[0m[0m
at org.apache.spark.scheduler.DAGScheduler.handleJobCancellation(DAGScheduler.scala:2918)[0m[0m
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleJobTagCancelled$7(DAGScheduler.scala:1266)[0m[0m
at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.scala:18)[0m[0m
at scala.collection.mutable.HashSet$Node.foreach(HashSet.scala:450)[0m[0m
at scala.collection.mutable.HashSet.foreach(HashSet.scala:376)[0m[0m
at org.apache.spark.scheduler.DAGScheduler.handleJobTagCancelled(DAGScheduler.scala:1266)[0m[0m
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3219)[0m[0m
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3194)[0m[0m
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3183)[0m[0m
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:50)[0m[0m
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1017)[0m[0m
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2489)[0m[0m
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2510)[0m[0m
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2529)[0m[0m
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2554)[0m[0m
at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1057)[0m[0m
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)[0m[0m
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)[0m[0m
at org.apache.spark.rdd.RDD.withScope(RDD.scala:417)[0m[0m
at org.apache.spark.rdd.RDD.collect(RDD.scala:1056)[0m[0m
at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:462)[0m[0m
at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.$anonfun$executeCollect$1(AdaptiveSparkPlanExec.scala:396)[0m[0m
at org.apache.spark.sql.execution.adaptive.ResultQueryStageExec.$anonfun$doMaterialize$1(QueryStageExec.scala:328)[0m[0m
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$4(SQLExecution.scala:348)[0m[0m
at org.apache.spark.sql.execution.SQLExecution$.withSessionTagsApplied(SQLExecution.scala:298)[0m[0m
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$3(SQLExecution.scala:346)[0m[0m
at org.apache.spark.JobArtifactSet$.withActiveJobArtifactState(JobArtifactSet.scala:94)[0m[0m
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$2(SQLExecution.scala:342)[0m[0m
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)[0m[0m
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)[0m[0m
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)[0m[0m
at java.base/java.lang.Thread.run(Thread.java:840)[0m[0m
```
According to the following log, the job was actually canceled twice in parallel mode.
```
Line 2: 2025-10-27T05:32:55.7283226Z 05:32:55.724 WARN org.apache.spark.sql.execution.SQLExecution: The SQLExecution is org.apache.spark.sql.execution.SQLExecution$@163503c1 and the executionIdJobTag is spark-session-42c4d99b-c576-4172-b5ed-56c61bb79f14-execution-root-id-25164
Line 10: 2025-10-27T05:32:55.7668804Z 05:32:55.765 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.0 in stage 29.0 (TID 56) (localhost executor driver): TaskKilled (Stage cancelled: [SPARK_JOB_CANCELLED] Job 20 cancelled part of cancelled job tags spark-session-42c4d99b-c576-4172-b5ed-56c61bb79f14-execution-root-id-25164 SQLSTATE: XXKDA)
Line 73: 2025-10-27T05:32:57.1915923Z [0m[[0m[0minfo[0m] [0m[0m[31m org.apache.spark.SparkException: [SPARK_JOB_CANCELLED] Job 20 cancelled part of cancelled job tags spark-session-42c4d99b-c576-4172-b5ed-56c61bb79f14-execution-root-id-25164 SQLSTATE: XXKDA[0m[0m
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2284078178
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -195,6 +195,10 @@ object SQLExecution extends Logging {
}
}
}
+
+ sparkSession.sparkContext.cancelJobsWithTag(
+ executionIdJobTag(sparkSession, executionId))
Review Comment:
nested query execution can also hit this code branch, or did I miss something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2501562881
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -135,97 +135,110 @@ object SQLExecution extends Logging {
withSQLConfPropagated(sparkSession) {
sparkSession.artifactManager.withResources {
- withSessionTagsApplied(sparkSession) {
- var ex: Option[Throwable] = None
- var isExecutedPlanAvailable = false
- val startTime = System.nanoTime()
- val startEvent = SparkListenerSQLExecutionStart(
- executionId = executionId,
- rootExecutionId = Some(rootExecutionId),
- description = desc,
- details = callSite.longForm,
- physicalPlanDescription = "",
- sparkPlanInfo = SparkPlanInfo.EMPTY,
- time = System.currentTimeMillis(),
- modifiedConfigs = redactedConfigs,
- jobTags = sc.getJobTags(),
- jobGroupId = Option(sc.getLocalProperty(SparkContext.SPARK_JOB_GROUP_ID))
- )
- try {
- body match {
- case Left(e) =>
- sc.listenerBus.post(startEvent)
+ synchronized {
Review Comment:
The value of `SPARK_JOB_TAGS` affects the job cancellation logic [here](https://github.com/apache/spark/blob/9894abcca68aa9339a740c676d1656f24ca3cedc/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1258-L1262). And the SQ method adds and removes job tags for each thread [here](https://github.com/apache/spark/blob/9894abcca68aa9339a740c676d1656f24ca3cedc/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala#L279C20-L287), but when multiple threads share the same SparkContext, the value may be overwritten or become inconsistent. Therefore, we introduce a lock here to ensure thread-safe isolation of the tag values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf closed pull request #52039: [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks.
URL: https://github.com/apache/spark/pull/52039
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf closed pull request #52039: [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks.
URL: https://github.com/apache/spark/pull/52039
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2505651795
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -135,97 +135,110 @@ object SQLExecution extends Logging {
withSQLConfPropagated(sparkSession) {
sparkSession.artifactManager.withResources {
- withSessionTagsApplied(sparkSession) {
- var ex: Option[Throwable] = None
- var isExecutedPlanAvailable = false
- val startTime = System.nanoTime()
- val startEvent = SparkListenerSQLExecutionStart(
- executionId = executionId,
- rootExecutionId = Some(rootExecutionId),
- description = desc,
- details = callSite.longForm,
- physicalPlanDescription = "",
- sparkPlanInfo = SparkPlanInfo.EMPTY,
- time = System.currentTimeMillis(),
- modifiedConfigs = redactedConfigs,
- jobTags = sc.getJobTags(),
- jobGroupId = Option(sc.getLocalProperty(SparkContext.SPARK_JOB_GROUP_ID))
- )
- try {
- body match {
- case Left(e) =>
- sc.listenerBus.post(startEvent)
+ synchronized {
Review Comment:
the spark local property is a thread local, are you sure the value can be changed by other threads?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2473467044
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLExecutionSuite.scala:
##########
@@ -50,7 +50,7 @@ class SQLExecutionSuite extends SparkFunSuite with SQLConfHelper {
}
}
- test("concurrent query execution with fork-join pool (SPARK-13747)") {
+ ignore("concurrent query execution with fork-join pool (SPARK-13747)") {
Review Comment:
The first message indicates a task being killed due to job group cancellation. The second message indicates a job is cancelled due to job group cancellation. It does not indicate we cancel the same job group twice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2501568866
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -135,97 +135,110 @@ object SQLExecution extends Logging {
withSQLConfPropagated(sparkSession) {
sparkSession.artifactManager.withResources {
- withSessionTagsApplied(sparkSession) {
- var ex: Option[Throwable] = None
- var isExecutedPlanAvailable = false
- val startTime = System.nanoTime()
- val startEvent = SparkListenerSQLExecutionStart(
- executionId = executionId,
- rootExecutionId = Some(rootExecutionId),
- description = desc,
- details = callSite.longForm,
- physicalPlanDescription = "",
- sparkPlanInfo = SparkPlanInfo.EMPTY,
- time = System.currentTimeMillis(),
- modifiedConfigs = redactedConfigs,
- jobTags = sc.getJobTags(),
- jobGroupId = Option(sc.getLocalProperty(SparkContext.SPARK_JOB_GROUP_ID))
- )
- try {
- body match {
- case Left(e) =>
- sc.listenerBus.post(startEvent)
+ synchronized {
Review Comment:
@cloud-fan @wangyum Please help to take a look. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289] Invoke cancelJobGroup before start passing the ExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on PR #52039:
URL: https://github.com/apache/spark/pull/52039#issuecomment-3190886389
@wangyum @cloud-fan Could you help to review this PR? Thanks for your help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobGroup before start passing the ExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2281081011
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -110,6 +110,8 @@ object SQLExecution extends Logging {
redactedStr.substring(0, Math.min(truncateLength, redactedStr.length))
}.getOrElse(callSite.shortForm)
+ sparkSession.sparkContext.setJobGroup(executionId.toString, desc, true)
Review Comment:
Isn't this already being dealt with tags above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on PR #52039:
URL: https://github.com/apache/spark/pull/52039#issuecomment-3199199407
I couldn't reproduce the failed unit tests from CI — they pass in my local environment.
```
2025-08-18T06:33:50.1460342Z [0m[[0m[0minfo[0m] [0m[0m[31m- concurrent query execution with fork-join pool (SPARK-13747) *** FAILED *** (1 second, 617 milliseconds)[0m[0m
2025-08-18T06:33:50.1463536Z [0m[[0m[0minfo[0m] [0m[0m[31m org.apache.spark.SparkException: [SPARK_JOB_CANCELLED] Job 13 cancelled part of cancelled job tags spark-session-bf217928-ec88-4b87-8190-490644c0873f-execution-root-id-23546 SQLSTATE: XXKDA[0m[0m
2025-08-18T06:33:50.1466074Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.errors.SparkCoreErrors$.sparkJobCancelled(SparkCoreErrors.scala:222)[0m[0m
2025-08-18T06:33:50.1470809Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGScheduler.handleJobCancellation(DAGScheduler.scala:2865)[0m[0m
2025-08-18T06:33:50.1472812Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleJobTagCancelled$7(DAGScheduler.scala:1258)[0m[0m
2025-08-18T06:33:50.1474859Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.scala:18)[0m[0m
2025-08-18T06:33:50.1495157Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.mutable.HashSet$Node.foreach(HashSet.scala:450)[0m[0m
2025-08-18T06:33:50.1499692Z [0m[[0m[0minfo[0m] [0m[0m[31m at scala.collection.mutable.HashSet.foreach(HashSet.scala:376)[0m[0m
2025-08-18T06:33:50.1504374Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGScheduler.handleJobTagCancelled(DAGScheduler.scala:1258)[0m[0m
2025-08-18T06:33:50.1507546Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3166)[0m[0m
2025-08-18T06:33:50.1510448Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3141)[0m[0m
2025-08-18T06:33:50.1519005Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3130)[0m[0m
2025-08-18T06:33:50.1546475Z [0m[[0m[0minfo[0m] [0m[0m[31m at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:50)[0m[0m
2025-08-18T06:33:50.2622276Z [0m[[0m[0minfo[0m] [0m[0m[32m- Finding QueryExecution for given executionId (106 milliseconds)[0m[0m
```
@cloud-fan Do you have any input? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobGroup before start passing the ExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #52039:
URL: https://github.com/apache/spark/pull/52039#issuecomment-3196313145
> This PR ensures that the task is properly canceled ...
How is this possible? Driver can only send an RPC to kill a task, which is not guaranteed to success. The only reliable way to is to shut down the corresponding executors forciably, which is the job of task reaper (SPARK-18761).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobGroup before start passing the ExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2281150672
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -110,6 +110,8 @@ object SQLExecution extends Logging {
redactedStr.substring(0, Math.min(truncateLength, redactedStr.length))
}.getOrElse(callSite.shortForm)
+ sparkSession.sparkContext.setJobGroup(executionId.toString, desc, true)
Review Comment:
@HyukjinKwon Thanks for your review.
You're right that tags help with tracking, but `setJobGroup ` enables cancellation of all jobs in the group, which can be useful for interrupting queries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2284080835
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -195,6 +195,10 @@ object SQLExecution extends Logging {
}
}
}
+
+ sparkSession.sparkContext.cancelJobsWithTag(
+ executionIdJobTag(sparkSession, executionId))
Review Comment:
I think the way we clean up shuffles looks better. AQE will track all the query stages executed, so that we can clean up shuffle as soon as a query (whether nested or not) is completed. But here we use job group, which is only set for the root execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2283983312
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -195,6 +195,10 @@ object SQLExecution extends Logging {
}
}
}
+
+ sparkSession.sparkContext.cancelJobsWithTag(
+ executionIdJobTag(sparkSession, executionId))
Review Comment:
> We should only do this if this is the root execution
It seems that `EXECUTION_ROOT_ID_KEY `is already set to executionId [here](https://github.com/JkSelf/spark/blob/f44a0e861534a629bbd68973aeb43aceebe4cd16/sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala#L86-L89), so there's no need to add this check, right?
>BTW does it work for AQE?
Yes. I re-ran the test with AQE enabled, and it worked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2462829841
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLExecutionSuite.scala:
##########
@@ -50,7 +50,7 @@ class SQLExecutionSuite extends SparkFunSuite with SQLConfHelper {
}
}
- test("concurrent query execution with fork-join pool (SPARK-13747)") {
+ ignore("concurrent query execution with fork-join pool (SPARK-13747)") {
Review Comment:
I'm a bit confused. They are individual queries with different execution ids, right? How can cancelling one break the other?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on PR #52039:
URL: https://github.com/apache/spark/pull/52039#issuecomment-3789564802
@cloud-fan @HyukjinKwon @wangyum @zhouyuan
We have fix this issue in gluten side and no need this change. Thanks all for your help to investigate this issue. Will close this PR. Thanks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobGroup before start passing the ExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #52039:
URL: https://github.com/apache/spark/pull/52039#issuecomment-3195914871
Let's fix up the PR title, and describe the background in the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2454046416
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLExecutionSuite.scala:
##########
@@ -50,7 +50,7 @@ class SQLExecutionSuite extends SparkFunSuite with SQLConfHelper {
}
}
- test("concurrent query execution with fork-join pool (SPARK-13747)") {
+ ignore("concurrent query execution with fork-join pool (SPARK-13747)") {
Review Comment:
@cloud-fan
I looked deeper into the failed unit tests and found that they involve multiple threads sharing the same SparkContext.
If a job is cancelled in one thread, it can cause exceptions in other threads that are still running. How should we handle this concurrent scenario? Would it be acceptable to skip the cancel operation for this test case, or do you have other suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2454046416
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLExecutionSuite.scala:
##########
@@ -50,7 +50,7 @@ class SQLExecutionSuite extends SparkFunSuite with SQLConfHelper {
}
}
- test("concurrent query execution with fork-join pool (SPARK-13747)") {
+ ignore("concurrent query execution with fork-join pool (SPARK-13747)") {
Review Comment:
@cloud-fan
I looked deeper into the failed unit tests and found that they involve multiple threads sharing the same SparkContext.
If a job is cancelled in one thread, it can cause exceptions in other threads that are still running. How should we handle this concurrent scenario? Would it be acceptable to skip this test case, or do you have other suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53289][SQL] Invoke cancelJobsWithTag before start passing the SparkListenerSQLExecutionEnd event to prevent execution of canceled tasks. [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #52039:
URL: https://github.com/apache/spark/pull/52039#discussion_r2468356524
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLExecutionSuite.scala:
##########
@@ -50,7 +50,7 @@ class SQLExecutionSuite extends SparkFunSuite with SQLConfHelper {
}
}
- test("concurrent query execution with fork-join pool (SPARK-13747)") {
+ ignore("concurrent query execution with fork-join pool (SPARK-13747)") {
Review Comment:
`The SQLExecution is ... and the executionIdJobTag is ...` only appear once, why do you think it's canceled twice?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org