You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ganesha S (Jira)" <ji...@apache.org> on 2026/08/06 08:52:00 UTC
[jira] [Updated] (SPARK-58616) Attribute a reason for context-wide job cancellation in cancelAllJobs
[ https://issues.apache.org/jira/browse/SPARK-58616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ganesha S updated SPARK-58616:
------------------------------
Description:
When a context-wide cancellation fires, every job caught by it fails with the fixed message:
{noformat}
[SPARK_JOB_CANCELLED] Job <id> cancelled as part of cancellation of all jobs
{noformat}
That says what happened, but not who triggered it or why. It is also reported dentically on jobs that were perfectly healthy and are only being aborted as collateral for someone else's cancellation, so when reading logs, a context-wide cancel cannot be distinguished from a genuine failure of the job in question. This makes investigating this class of failure slow.
The sibling cancellation APIs already solve this: SPARK-19549 added an optional reason to {{cancelJob}} / {{{}cancelStage{}}}, and SPARK-48900 added one to {{cancelJobGroup}} and {{{}cancelJobsWithTag{}}}. {{cancelAllJobs}} is the remaining cancellation entry point with no way to attribute intent, and, unlike its siblings, it does not even log its invocation.
was:
When a context-wide cancellation fires, every job caught by it fails with the fixed
message:
{noformat}
[SPARK_JOB_CANCELLED] Job <id> cancelled as part of cancellation of all jobs
{noformat}
That says what happened, but not who triggered it or why. It is also reported dentically on jobs that were perfectly healthy and are only being aborted as collateral for someone else's cancellation, so when reading logs, a context-wide cancel cannot be distinguished from a genuine failure of the job in question. This makes investigating this class of failure slow.
The sibling cancellation APIs already solve this: SPARK-19549 added an optional reason to \{{cancelJob}} / \{{cancelStage}}, and SPARK-48900 added one to \{{cancelJobGroup}} and \{{cancelJobsWithTag}}. \{{cancelAllJobs}} is the remaining cancellation entry point with no way to attribute intent, and, unlike its siblings, it does not even log its invocation.
> Attribute a reason for context-wide job cancellation in cancelAllJobs
> ---------------------------------------------------------------------
>
> Key: SPARK-58616
> URL: https://issues.apache.org/jira/browse/SPARK-58616
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.3.0
> Reporter: Ganesha S
> Priority: Major
>
> When a context-wide cancellation fires, every job caught by it fails with the fixed message:
> {noformat}
> [SPARK_JOB_CANCELLED] Job <id> cancelled as part of cancellation of all jobs
> {noformat}
> That says what happened, but not who triggered it or why. It is also reported dentically on jobs that were perfectly healthy and are only being aborted as collateral for someone else's cancellation, so when reading logs, a context-wide cancel cannot be distinguished from a genuine failure of the job in question. This makes investigating this class of failure slow.
> The sibling cancellation APIs already solve this: SPARK-19549 added an optional reason to {{cancelJob}} / {{{}cancelStage{}}}, and SPARK-48900 added one to {{cancelJobGroup}} and {{{}cancelJobsWithTag{}}}. {{cancelAllJobs}} is the remaining cancellation entry point with no way to attribute intent, and, unlike its siblings, it does not even log its invocation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org