Help understanding spark.task.maxFailures

[Yana Kadiyska <ya...@gmail.com>]

Hi community, this one should be an easy one:

I have left spark.task.maxFailures to it's default (which should be
4). I see a job that shows the following statistics for Tasks:
Succeeded/Total

7109/819 (1 failed)

So there were 819 tasks to start with. I have 2 executors in that
cluster. From Spark docs it says spark.task.maxFailures is the number
of times to try a task before a job is given up. So I was imagining
that 819*4 (i.e. 3276) would be the max number to ever see in the
succeeded (accounting for retries on every possibly task). even that
3276*2 (6552, if it's per task per executor) does not account for 7109
successfull tasks.

Could anyone help explain why I'm seeing such high number of succeeded tasks?

Re: Help understanding spark.task.maxFailures

[Mayur Rustagi <ma...@gmail.com>]

stragglers?

Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>



On Tue, Jul 1, 2014 at 12:40 AM, Yana Kadiyska <ya...@gmail.com>
wrote:

> Hi community, this one should be an easy one:
>
> I have left spark.task.maxFailures to it's default (which should be
> 4). I see a job that shows the following statistics for Tasks:
> Succeeded/Total
>
> 7109/819 (1 failed)
>
> So there were 819 tasks to start with. I have 2 executors in that
> cluster. From Spark docs it says spark.task.maxFailures is the number
> of times to try a task before a job is given up. So I was imagining
> that 819*4 (i.e. 3276) would be the max number to ever see in the
> succeeded (accounting for retries on every possibly task). even that
> 3276*2 (6552, if it's per task per executor) does not account for 7109
> successfull tasks.
>
> Could anyone help explain why I'm seeing such high number of succeeded
> tasks?
>