You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by praveen S <my...@gmail.com> on 2015/10/26 12:26:59 UTC
Concurrent execution of actions within a driver
Does spark run different actions of an rdd within a driver in parallel
also?
Let's say
class Driver{
val rdd1= sc. textFile("... ")
val rdd2=sc.textFile("....")
rdd1. collect //Action 1
rdd2. collect //Action 2
}
Does Spark run Action 1 & 2 run in parallel? ( some kind of a pass through
the driver code and than start the execution)?
if not than is using threads safe for independent actions/red's?
Re: Concurrent execution of actions within a driver
Posted by Rishitesh Mishra <rm...@snappydata.io>.
Spark executes tasks on an action. An action is broken down to multiple
tasks. Multiple tasks from different actions run either in FIFO or FAIR
mode depending on spark.scheduler.mode.
Of course to get benefit of FAIR scheduling the two actions should be
called by different threads.
On Mon, Oct 26, 2015 at 5:01 PM, Fengdong Yu <fe...@everstring.com>
wrote:
> not parallel.
>
> Spark only execute tasks with Action,(‘collect' here)
>
> rdd1.collect and rdd2.collect are executed sequencely, so Spark execute
> two tasks one by one.
>
>
>
>
> On Oct 26, 2015, at 7:26 PM, praveen S <my...@gmail.com> wrote:
>
> Does spark run different actions of an rdd within a driver in parallel
> also?
>
> Let's say
> class Driver{
>
> val rdd1= sc. textFile("... ")
> val rdd2=sc.textFile("....")
> rdd1. collect //Action 1
> rdd2. collect //Action 2
>
> }
>
> Does Spark run Action 1 & 2 run in parallel? ( some kind of a pass through
> the driver code and than start the execution)?
>
> if not than is using threads safe for independent actions/red's?
>
>
>
--
Regards,
Rishitesh Mishra,
SnappyData . (http://www.snappydata.io/)
https://in.linkedin.com/in/rishiteshmishra
Re: Concurrent execution of actions within a driver
Posted by Fengdong Yu <fe...@everstring.com>.
not parallel.
Spark only execute tasks with Action,(‘collect' here)
rdd1.collect and rdd2.collect are executed sequencely, so Spark execute two tasks one by one.
> On Oct 26, 2015, at 7:26 PM, praveen S <my...@gmail.com> wrote:
>
> Does spark run different actions of an rdd within a driver in parallel also?
>
> Let's say
> class Driver{
>
> val rdd1= sc. textFile("... ")
> val rdd2=sc.textFile("....")
> rdd1. collect //Action 1
> rdd2. collect //Action 2
>
> }
>
> Does Spark run Action 1 & 2 run in parallel? ( some kind of a pass through the driver code and than start the execution)?
>
> if not than is using threads safe for independent actions/red's?
>
RE: Concurrent execution of actions within a driver
Posted by Silvio Fiorito <si...@granturing.com>.
There is a collectAsync action if you want to run them in parallel, but keep in mind the two jobs will need to share resources and you should use the FAIR scheduler.
________________________________
From: praveen S<ma...@gmail.com>
Sent: 10/26/2015 4:27 AM
To: user@spark.apache.org<ma...@spark.apache.org>
Subject: Concurrent execution of actions within a driver
Does spark run different actions of an rdd within a driver in parallel also?
Let's say
class Driver{
val rdd1= sc. textFile("... ")
val rdd2=sc.textFile("....")
rdd1. collect //Action 1
rdd2. collect //Action 2
}
Does Spark run Action 1 & 2 run in parallel? ( some kind of a pass through the driver code and than start the execution)?
if not than is using threads safe for independent actions/red's?