You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Shushant Arora <sh...@gmail.com> on 2016/02/18 08:49:26 UTC

spark stages in parallel

can two stages of single job run in parallel in spark?

e.g one stage is ,map transformation and another is repartition on mapped
rdd.

rdd.map(function,100).repartition(30);

can it happen that map transformation which is running 100 tasks after few
of them say (10 )  are finished and spark started another stage repartition
which started copying data from mapped stage nodes in parallel.

Thanks

Re: spark stages in parallel

Posted by Hemant Bhanawat <he...@gmail.com>.

Not possible as of today. See
https://issues.apache.org/jira/browse/SPARK-2387

Hemant Bhanawat
https://www.linkedin.com/in/hemant-bhanawat-92a3811
www.snappydata.io

On Thu, Feb 18, 2016 at 1:19 PM, Shushant Arora <sh...@gmail.com>
wrote:

> can two stages of single job run in parallel in spark?
>
> e.g one stage is ,map transformation and another is repartition on mapped
> rdd.
>
> rdd.map(function,100).repartition(30);
>
> can it happen that map transformation which is running 100 tasks after few
> of them say (10 )  are finished and spark started another stage repartition
> which started copying data from mapped stage nodes in parallel.
>
> Thanks
>