Can I set max execution time for any task in a job?

[Mohamed Lrhazi <Mo...@georgetown.edu>]

Is that possible, if not, how would one do it from PySpark ?

This probably does not make sense in most cases, but am writing a script
where my job involves downloading and pushing data into cassandra..
sometimes a task hangs forever, and I dont really mind killing it.. The job
is not actually computing some result that requires all tasks to succeed.

Thanks,
Mohamed.

Re: Can I set max execution time for any task in a job?

[Akhil Das <ak...@sigmoidanalytics.com>]

There is a spark listener interface
<https://spark.apache.org/docs/1.0.0/api/scala/index.html#org.apache.spark.scheduler.SparkListener>
which
can be used to trigger events like jobStarted, TaskGotResults etc but i
don't think you can set execution time anywhere. If a task is hung, its
mostly because of the GC pause (depends on your job), if you can paste the
code, then probably we can tell you where the bottleneck is.

Thanks
Best Regards

On Tue, Dec 16, 2014 at 9:59 AM, Mohamed Lrhazi <
Mohamed.Lrhazi@georgetown.edu> wrote:
>
> Is that possible, if not, how would one do it from PySpark ?
>
> This probably does not make sense in most cases, but am writing a script
> where my job involves downloading and pushing data into cassandra..
> sometimes a task hangs forever, and I dont really mind killing it.. The job
> is not actually computing some result that requires all tasks to succeed.
>
> Thanks,
> Mohamed.
>