SparkContext#cancelJobGroup : is it safe? Who got burn? Who is alive?

[Bertrand Dechoux <de...@gmail.com>]

Hi,

I am wondering about the safety of the *SparkContext#cancelJobGroup* method
that should allow to stop specific (ie not all) jobs inside a spark context.

There is a big disclaimer (
https://spark.apache.org/docs/1.6.0/api/java/org/apache/spark/SparkContext.html#setJobGroup(java.lang.String,%20java.lang.String,%20boolean)
.

If interruptOnCancel is set to true for the job group, then job
> cancellation will result in Thread.interrupt() being called on the job's
> executor threads. This is useful to help ensure that the tasks are actually
> stopped in a timely manner, but is off by default due to HDFS-1208, where
> HDFS may respond to Thread.interrupt() by marking nodes as dead.


I have two main questions :

   1. What is the expected behavior if it is not interrupted on cancel? I
   am especially curious about the YARN case with HDFS but any info is welcome.
   2. Who is or was using the *interruptOnCancel* ? Do you got burn? It is
   still working without any incident?

Thanks in advance for any info, feedbacks and war stories.

Bertrand Dechoux