You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Adam McElwee <ad...@mcelwee.me> on 2016/01/22 19:15:41 UTC

Disable speculative retry only for specific stages?

I've used speculative execution a couple times in the past w/ good results,
but I have one stage in my job with a non-idempotent operation in a
`forEachPartition` block. I don't see a way to disable speculative retry on
certain stages, but does anyone know of any tricks to help out here?

Spark version: 1.6.0

Thanks,
-Adam

Re: Disable speculative retry only for specific stages?

Posted by Ted Yu <yu...@gmail.com>.

Looked at:
https://spark.apache.org/docs/latest/configuration.html

I don't think Spark supports per stage speculation.

On Fri, Jan 22, 2016 at 10:15 AM, Adam McElwee <ad...@mcelwee.me> wrote:

> I've used speculative execution a couple times in the past w/ good
> results, but I have one stage in my job with a non-idempotent operation in
> a `forEachPartition` block. I don't see a way to disable speculative retry
> on certain stages, but does anyone know of any tricks to help out here?
>
> Spark version: 1.6.0
>
> Thanks,
> -Adam
>