[jira] [Commented] (SPARK-43952) Cancel Spark jobs not only by a single "jobgroup", but allow multiple "job tags"

["Juliusz Sompolski (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-43952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17728791#comment-17728791 ] 

Juliusz Sompolski commented on SPARK-43952:
-------------------------------------------

Indirectly related with https://issues.apache.org/jira/browse/SPARK-43754 to allow Spark Connect cancellation of queries not conflict with other places setting job groups.

> Cancel Spark jobs not only by a single "jobgroup", but allow multiple "job tags"
> --------------------------------------------------------------------------------
>
>                 Key: SPARK-43952
>                 URL: https://issues.apache.org/jira/browse/SPARK-43952
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 3.5.0
>            Reporter: Juliusz Sompolski
>            Priority: Major
>
> Currently, the only way to cancel running Spark Jobs is by using SparkContext.cancelJobGroup, using a job group name that was previously set using SparkContext.setJobGroup. This is problematic if multiple different parts of the system want to do cancellation, and set their own ids.
> For example, [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala#L133] sets it's own job group, which may override job group set by user. This way, if user cancels the job group they set, it will not cancel these broadcast jobs launches from within their jobs...
> As a solution, consider adding SparkContext.addJobTag / SparkContext.removeJobTag, which would allow to have multiple "tags" on the jobs, and introduce SparkContext.cancelJobsByTag to allow more flexible cancelling of jobs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org