You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "panbingkun (via GitHub)" <gi...@apache.org> on 2024/03/26 11:28:54 UTC

[PR] [MINOR][CORE] When failed to canceling the job group, add a warning log [spark]

panbingkun opened a new pull request, #45722:
URL: https://github.com/apache/spark/pull/45722

   ### What changes were proposed in this pull request?
   The pr aims to add add a warning log when failed to canceling the job group.
   
   
   ### Why are the changes needed?
   In production, we found that when there are many jobs, if we specify `a just-submitted job group` and `cancel` it using the API `SparkContext#cancelJobGroup(groupId: String)`, the cancellation will fail (`silently`) because the job has not yet entered `activeJobs`, and there is `no warning` prompt.
   (PS: Due to historical reasons, we are not currently using the newer API `SparkContext#cancelJobGroupAndFutureJobs(groupId: String)`)
   
   I propose to add `a warning log` here to help troubleshoot `issues`.
   
   ### Does this PR introduce _any_ user-facing change?
   Yes, only for log.
   
   
   ### How was this patch tested?
   N/A
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [MINOR][CORE] When failed to canceling the job group, add a warning log [spark]

Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #45722:
URL: https://github.com/apache/spark/pull/45722#issuecomment-2020176229

   cc @cloud-fan @HyukjinKwon 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [MINOR][CORE] When failed to canceling the job group, add a warning log [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #45722:
URL: https://github.com/apache/spark/pull/45722#discussion_r1539162989


##########
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##########
@@ -1220,6 +1220,10 @@ private[spark] class DAGScheduler(
         _.getProperty(SparkContext.SPARK_JOB_GROUP_ID) == groupId
       }
     }
+    if (activeInGroup.isEmpty && !cancelFutureJobs) {
+      logWarning(s"Cancel job group $groupId failed due to " +

Review Comment:
   `Failed to cancel job group $groupId. Cannot find active jobs for it.`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [MINOR][CORE] When failed to canceling the job group, add a warning log [spark]

Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #45722:
URL: https://github.com/apache/spark/pull/45722#discussion_r1539192011


##########
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##########
@@ -1220,6 +1220,10 @@ private[spark] class DAGScheduler(
         _.getProperty(SparkContext.SPARK_JOB_GROUP_ID) == groupId
       }
     }
+    if (activeInGroup.isEmpty && !cancelFutureJobs) {
+      logWarning(s"Cancel job group $groupId failed due to " +

Review Comment:
   Done.
   Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [MINOR][CORE] When failed to canceling the job group, add a warning log [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45722:
URL: https://github.com/apache/spark/pull/45722#issuecomment-2020977543

   Merged to master. Thank you all.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [MINOR][CORE] When failed to canceling the job group, add a warning log [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #45722: [MINOR][CORE] When failed to canceling the job group, add a warning log
URL: https://github.com/apache/spark/pull/45722


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org