Possible to make one executor be able to work on multiple tasks simultaneously?

[Victor Tso-Guillen <vt...@paxata.com>]

I'm thinking of local mode where multiple virtual executors occupy the same
vm. Can we have the same configuration in spark standalone cluster mode?

Re: Possible to make one executor be able to work on multiple tasks simultaneously?

[Victor Tso-Guillen <vt...@paxata.com>]

Any more thoughts on this? I'm not sure how to do this yet.


On Fri, Aug 29, 2014 at 12:10 PM, Victor Tso-Guillen <vt...@paxata.com>
wrote:

> Standalone. I'd love to tell it that my one executor can simultaneously
> serve, say, 16 tasks at once for an arbitrary number of distinct jobs.
>
>
> On Fri, Aug 29, 2014 at 11:29 AM, Matei Zaharia <ma...@gmail.com>
> wrote:
>
>> Yes, executors run one task per core of your machine by default. You can
>> also manually launch them with more worker threads than you have cores.
>> What cluster manager are you on?
>>
>> Matei
>>
>> On August 29, 2014 at 11:24:33 AM, Victor Tso-Guillen (vtso@paxata.com)
>> wrote:
>>
>>  I'm thinking of local mode where multiple virtual executors occupy the
>> same vm. Can we have the same configuration in spark standalone cluster
>> mode?
>>
>>
>

Re: Possible to make one executor be able to work on multiple tasks simultaneously?

[Victor Tso-Guillen <vt...@paxata.com>]

I'm pretty sure the issue was an interaction with another subsystem. Thanks
for your patience with me!


On Tue, Sep 2, 2014 at 10:05 AM, Sean Owen <so...@cloudera.com> wrote:

> +user@
>
> An executor is specific to an application, but an application can be
> executing many jobs at once. So as I understand many jobs' tasks can
> be executing at once on an executor.
>
> You may not use your full 80-way parallelism if, for example, your
> data set doesn't have 80 partitions. I also believe Spark will not
> necessarily spread the load over executors, instead preferring to
> respect data and rack locality if possible. Those are two reasons you
> might see only 4 executors active. If you mean only 4 executors exist
> at all, is it possible the other 4 can't provide the memory you're
> asking for?
>
>
> On Tue, Sep 2, 2014 at 5:56 PM, Victor Tso-Guillen <vt...@paxata.com>
> wrote:
> > Actually one more question, since in preliminary runs I wasn't sure if I
> > understood what's going on. Are the cores allocated to an executor able
> to
> > execute tasks for different jobs simultaneously, or just for one job at a
> > time? I have 10 workers with 8 cores each, and it appeared that one job
> got
> > four executors at once, then four more later on. The system wasn't
> anywhere
> > near saturation of 80 cores so I would've expected all 8 cores to be
> running
> > simultaneously.
> >
> > If there's value to these questions, please reply back to the list.
> >
> >
> > On Tue, Sep 2, 2014 at 6:58 AM, Victor Tso-Guillen <vt...@paxata.com>
> wrote:
> >>
> >> Thank you for the help, guys. So as I expected, I didn't fully
> understand
> >> the options. I had SPARK_WORKER_CORES set to 1 because I did not realize
> >> that by setting to > 1 it would mean an executor could operate on
> multiple
> >> tasks simultaneously. I just thought it was a hint to Spark that that
> >> executor could be expected to use that many threads, but otherwise I
> had not
> >> understood that it affected the scheduler that way. Thanks!
> >>
> >>
> >> On Sun, Aug 31, 2014 at 9:28 PM, Matei Zaharia <matei.zaharia@gmail.com
> >
> >> wrote:
> >>>
> >>>
> >>> Hey Victor,
> >>>
> >>> As Sean said, executors actually execute multiple tasks at a time. The
> >>> only reasons they wouldn't are either (1) if you launched an executor
> with
> >>> just 1 core (you can configure how many cores the executors will use
> when
> >>> you set up your Worker, or it will look at your system by default) or
> (2) if
> >>> your tasks are acquiring some kind of global lock, so only one can run
> at a
> >>> time.
> >>>
> >>> To test this, do the following:
> >>> - Launch your standalone cluster (you can do it on just one machine by
> >>> adding just "localhost" in the slaves file)
> >>> - Go to http://:4040 and look at the worker list. Do you see workers
> with
> >>> more than 1 core? If not, you need to launch the workers by hand or set
> >>> SPARK_WORKER_CORES in conf/spark-env.sh.
> >>> - Run your application. Make sure it has enough pending tasks for your
> >>> cores in the driver web UI (http://:4040), and if so, jstack one of
> the
> >>> CoarseGrainedExecutor processes on a worker to see what the threads are
> >>> doing. (Look for threads that contain TaskRunner.run in them)
> >>>
> >>> You can also try a simple CPU-bound job that launches lots of tasks
> like
> >>> this to see that all cores are being used:
> >>>
> >>> sc.parallelize(1 to 1000, 1000).map(_ => (1 to
> >>> 2000000000).product).count()
> >>>
> >>> Each task here takes 1-2 seconds to execute and there are 1000 of them
> so
> >>> it should fill up your cluster.
> >>>
> >>> Matei
> >>>
> >>>
> >>>
> >>> On August 31, 2014 at 9:18:02 PM, Victor Tso-Guillen
> >>> (vtso@paxata.com(mailto:vtso@paxata.com)) wrote:
> >>>
> >>> > I'm pretty sure my terminology matches that doc except the doc makes
> no
> >>> > explicit mention of machines. In standalone mode, you can spawn
> multiple
> >>> > workers on a single machine and each will babysit one executor (per
> >>> > application). In my observation as well each executor can be
> assigned many
> >>> > tasks but operates on one at a time. If there's a way to have it
> execute in
> >>> > multiple tasks simultaneously in a single VM can you please show me
> how?
> >>> > Maybe I'm missing the requisite configuration options, no matter how
> common
> >>> > or trivial...
> >>> >
> >>> > On Sunday, August 31, 2014, Sean Owen wrote:
> >>> > > The confusion may be your use of 'worker', which isn't matching
> what
> >>> > > 'worker' means in Spark. Have a look at
> >>> > > https://spark.apache.org/docs/latest/cluster-overview.html Of
> course
> >>> > > one VM can run many tasks at once; that's already how Spark works.
> >>> > >
> >>> > > On Sun, Aug 31, 2014 at 4:52 AM, Victor Tso-Guillen wrote:
> >>> > > > I might not be making myself clear, so sorry about that. I
> >>> > > > understand that a
> >>> > > > machine can have as many spark workers as you'd like, for example
> >>> > > > one per
> >>> > > > core. A worker may be assigned to a pool for one or more
> >>> > > > applications, but
> >>> > > > for a single application let's just say a single worker will have
> >>> > > > at most a
> >>> > > > single executor. An executor can be assigned multiple tasks in
> its
> >>> > > > queue,
> >>> > > > but will work on one task at a time only.
> >>> > > >
> >>> > > > In local mode, you can specify the number of executors you want
> and
> >>> > > > they
> >>> > > > will all reside in the same vm. Those executors will each be able
> >>> > > > to operate
> >>> > > > on a single task at a time, though they may also have an
> arbitrary
> >>> > > > number of
> >>> > > > tasks in their queue. From the standpoint of a vm, however, a vm
> >>> > > > can
> >>> > > > therefore operate on multiple tasks simultaneously in local mode.
> >>> > > >
> >>> > > > What I want is something similar in standalone mode (or mesos or
> >>> > > > YARN if
> >>> > > > that's the only way to do it) whereby I can have a single
> executor
> >>> > > > vm handle
> >>> > > > many tasks concurrently. Is this possible? Is my problem
> statement
> >>> > > > clear? If
> >>> > > > there's a misconception on my part on the deployment of a spark
> >>> > > > cluster I'd
> >>> > > > like to know it, but as of currently what we have deployed is
> like
> >>> > > > my first
> >>> > > > paragraph.
> >>> > > >
> >>> > > >
> >>> > > > On Sat, Aug 30, 2014 at 1:58 AM, Sean Owen wrote:
> >>> > > >>
> >>> > > >> A machine should have one worker, and many executors per worker
> >>> > > >> (one per
> >>> > > >> app). An executor runs many tasks. This is how it works for me
> in
> >>> > > >> standalone
> >>> > > >> mode at least!
> >>> > > >>
> >>> > > >> On Aug 30, 2014 3:08 AM, "Victor Tso-Guillen" wrote:
> >>> > > >>>
> >>> > > >>> A machine has many workers and a worker has an executor. I want
> >>> > > >>> the
> >>> > > >>> executor to handle many tasks at once, like in local mode.
> >>> > > >>>
> >>> > > >>>
> >>> > > >>> On Fri, Aug 29, 2014 at 5:51 PM, Sean Owen wrote:
> >>> > > >>>>
> >>> > > >>>> Hm, do you mean worker? Spark certainly works on many tasks
> per
> >>> > > >>>> machine
> >>> > > >>>> at once.
> >>> > > >>>>
> >>> > > >>>> On Aug 29, 2014 8:11 PM, "Victor Tso-Guillen" wrote:
> >>> > > >>>>>
> >>> > > >>>>> Standalone. I'd love to tell it that my one executor can
> >>> > > >>>>> simultaneously
> >>> > > >>>>> serve, say, 16 tasks at once for an arbitrary number of
> >>> > > >>>>> distinct jobs.
> >>> > > >>>>>
> >>> > > >>>>>
> >>> > > >>>>> On Fri, Aug 29, 2014 at 11:29 AM, Matei Zaharia
> >>> > > >>>>> wrote:
> >>> > > >>>>>>
> >>> > > >>>>>> Yes, executors run one task per core of your machine by
> >>> > > >>>>>> default. You
> >>> > > >>>>>> can also manually launch them with more worker threads than
> >>> > > >>>>>> you have cores.
> >>> > > >>>>>> What cluster manager are you on?
> >>> > > >>>>>>
> >>> > > >>>>>> Matei
> >>> > > >>>>>>
> >>> > > >>>>>> On August 29, 2014 at 11:24:33 AM, Victor Tso-Guillen
> >>> > > >>>>>> (vtso@paxata.com(javascript:;)) wrote:
> >>> > > >>>>>>
> >>> > > >>>>>> I'm thinking of local mode where multiple virtual executors
> >>> > > >>>>>> occupy the
> >>> > > >>>>>> same vm. Can we have the same configuration in spark
> >>> > > >>>>>> standalone cluster
> >>> > > >>>>>> mode?
> >>> > > >>>>>
> >>> > > >>>>>
> >>> > > >>>
> >>> > > >
> >>>
> >>
> >
>

Re: Possible to make one executor be able to work on multiple tasks simultaneously?

[Sean Owen <so...@cloudera.com>]

+user@

An executor is specific to an application, but an application can be
executing many jobs at once. So as I understand many jobs' tasks can
be executing at once on an executor.

You may not use your full 80-way parallelism if, for example, your
data set doesn't have 80 partitions. I also believe Spark will not
necessarily spread the load over executors, instead preferring to
respect data and rack locality if possible. Those are two reasons you
might see only 4 executors active. If you mean only 4 executors exist
at all, is it possible the other 4 can't provide the memory you're
asking for?


On Tue, Sep 2, 2014 at 5:56 PM, Victor Tso-Guillen <vt...@paxata.com> wrote:
> Actually one more question, since in preliminary runs I wasn't sure if I
> understood what's going on. Are the cores allocated to an executor able to
> execute tasks for different jobs simultaneously, or just for one job at a
> time? I have 10 workers with 8 cores each, and it appeared that one job got
> four executors at once, then four more later on. The system wasn't anywhere
> near saturation of 80 cores so I would've expected all 8 cores to be running
> simultaneously.
>
> If there's value to these questions, please reply back to the list.
>
>
> On Tue, Sep 2, 2014 at 6:58 AM, Victor Tso-Guillen <vt...@paxata.com> wrote:
>>
>> Thank you for the help, guys. So as I expected, I didn't fully understand
>> the options. I had SPARK_WORKER_CORES set to 1 because I did not realize
>> that by setting to > 1 it would mean an executor could operate on multiple
>> tasks simultaneously. I just thought it was a hint to Spark that that
>> executor could be expected to use that many threads, but otherwise I had not
>> understood that it affected the scheduler that way. Thanks!
>>
>>
>> On Sun, Aug 31, 2014 at 9:28 PM, Matei Zaharia <ma...@gmail.com>
>> wrote:
>>>
>>>
>>> Hey Victor,
>>>
>>> As Sean said, executors actually execute multiple tasks at a time. The
>>> only reasons they wouldn't are either (1) if you launched an executor with
>>> just 1 core (you can configure how many cores the executors will use when
>>> you set up your Worker, or it will look at your system by default) or (2) if
>>> your tasks are acquiring some kind of global lock, so only one can run at a
>>> time.
>>>
>>> To test this, do the following:
>>> - Launch your standalone cluster (you can do it on just one machine by
>>> adding just "localhost" in the slaves file)
>>> - Go to http://:4040 and look at the worker list. Do you see workers with
>>> more than 1 core? If not, you need to launch the workers by hand or set
>>> SPARK_WORKER_CORES in conf/spark-env.sh.
>>> - Run your application. Make sure it has enough pending tasks for your
>>> cores in the driver web UI (http://:4040), and if so, jstack one of the
>>> CoarseGrainedExecutor processes on a worker to see what the threads are
>>> doing. (Look for threads that contain TaskRunner.run in them)
>>>
>>> You can also try a simple CPU-bound job that launches lots of tasks like
>>> this to see that all cores are being used:
>>>
>>> sc.parallelize(1 to 1000, 1000).map(_ => (1 to
>>> 2000000000).product).count()
>>>
>>> Each task here takes 1-2 seconds to execute and there are 1000 of them so
>>> it should fill up your cluster.
>>>
>>> Matei
>>>
>>>
>>>
>>> On August 31, 2014 at 9:18:02 PM, Victor Tso-Guillen
>>> (vtso@paxata.com(mailto:vtso@paxata.com)) wrote:
>>>
>>> > I'm pretty sure my terminology matches that doc except the doc makes no
>>> > explicit mention of machines. In standalone mode, you can spawn multiple
>>> > workers on a single machine and each will babysit one executor (per
>>> > application). In my observation as well each executor can be assigned many
>>> > tasks but operates on one at a time. If there's a way to have it execute in
>>> > multiple tasks simultaneously in a single VM can you please show me how?
>>> > Maybe I'm missing the requisite configuration options, no matter how common
>>> > or trivial...
>>> >
>>> > On Sunday, August 31, 2014, Sean Owen wrote:
>>> > > The confusion may be your use of 'worker', which isn't matching what
>>> > > 'worker' means in Spark. Have a look at
>>> > > https://spark.apache.org/docs/latest/cluster-overview.html Of course
>>> > > one VM can run many tasks at once; that's already how Spark works.
>>> > >
>>> > > On Sun, Aug 31, 2014 at 4:52 AM, Victor Tso-Guillen wrote:
>>> > > > I might not be making myself clear, so sorry about that. I
>>> > > > understand that a
>>> > > > machine can have as many spark workers as you'd like, for example
>>> > > > one per
>>> > > > core. A worker may be assigned to a pool for one or more
>>> > > > applications, but
>>> > > > for a single application let's just say a single worker will have
>>> > > > at most a
>>> > > > single executor. An executor can be assigned multiple tasks in its
>>> > > > queue,
>>> > > > but will work on one task at a time only.
>>> > > >
>>> > > > In local mode, you can specify the number of executors you want and
>>> > > > they
>>> > > > will all reside in the same vm. Those executors will each be able
>>> > > > to operate
>>> > > > on a single task at a time, though they may also have an arbitrary
>>> > > > number of
>>> > > > tasks in their queue. From the standpoint of a vm, however, a vm
>>> > > > can
>>> > > > therefore operate on multiple tasks simultaneously in local mode.
>>> > > >
>>> > > > What I want is something similar in standalone mode (or mesos or
>>> > > > YARN if
>>> > > > that's the only way to do it) whereby I can have a single executor
>>> > > > vm handle
>>> > > > many tasks concurrently. Is this possible? Is my problem statement
>>> > > > clear? If
>>> > > > there's a misconception on my part on the deployment of a spark
>>> > > > cluster I'd
>>> > > > like to know it, but as of currently what we have deployed is like
>>> > > > my first
>>> > > > paragraph.
>>> > > >
>>> > > >
>>> > > > On Sat, Aug 30, 2014 at 1:58 AM, Sean Owen wrote:
>>> > > >>
>>> > > >> A machine should have one worker, and many executors per worker
>>> > > >> (one per
>>> > > >> app). An executor runs many tasks. This is how it works for me in
>>> > > >> standalone
>>> > > >> mode at least!
>>> > > >>
>>> > > >> On Aug 30, 2014 3:08 AM, "Victor Tso-Guillen" wrote:
>>> > > >>>
>>> > > >>> A machine has many workers and a worker has an executor. I want
>>> > > >>> the
>>> > > >>> executor to handle many tasks at once, like in local mode.
>>> > > >>>
>>> > > >>>
>>> > > >>> On Fri, Aug 29, 2014 at 5:51 PM, Sean Owen wrote:
>>> > > >>>>
>>> > > >>>> Hm, do you mean worker? Spark certainly works on many tasks per
>>> > > >>>> machine
>>> > > >>>> at once.
>>> > > >>>>
>>> > > >>>> On Aug 29, 2014 8:11 PM, "Victor Tso-Guillen" wrote:
>>> > > >>>>>
>>> > > >>>>> Standalone. I'd love to tell it that my one executor can
>>> > > >>>>> simultaneously
>>> > > >>>>> serve, say, 16 tasks at once for an arbitrary number of
>>> > > >>>>> distinct jobs.
>>> > > >>>>>
>>> > > >>>>>
>>> > > >>>>> On Fri, Aug 29, 2014 at 11:29 AM, Matei Zaharia
>>> > > >>>>> wrote:
>>> > > >>>>>>
>>> > > >>>>>> Yes, executors run one task per core of your machine by
>>> > > >>>>>> default. You
>>> > > >>>>>> can also manually launch them with more worker threads than
>>> > > >>>>>> you have cores.
>>> > > >>>>>> What cluster manager are you on?
>>> > > >>>>>>
>>> > > >>>>>> Matei
>>> > > >>>>>>
>>> > > >>>>>> On August 29, 2014 at 11:24:33 AM, Victor Tso-Guillen
>>> > > >>>>>> (vtso@paxata.com(javascript:;)) wrote:
>>> > > >>>>>>
>>> > > >>>>>> I'm thinking of local mode where multiple virtual executors
>>> > > >>>>>> occupy the
>>> > > >>>>>> same vm. Can we have the same configuration in spark
>>> > > >>>>>> standalone cluster
>>> > > >>>>>> mode?
>>> > > >>>>>
>>> > > >>>>>
>>> > > >>>
>>> > > >
>>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Possible to make one executor be able to work on multiple tasks simultaneously?

[Victor Tso-Guillen <vt...@paxata.com>]

Standalone. I'd love to tell it that my one executor can simultaneously
serve, say, 16 tasks at once for an arbitrary number of distinct jobs.


On Fri, Aug 29, 2014 at 11:29 AM, Matei Zaharia <ma...@gmail.com>
wrote:

> Yes, executors run one task per core of your machine by default. You can
> also manually launch them with more worker threads than you have cores.
> What cluster manager are you on?
>
> Matei
>
> On August 29, 2014 at 11:24:33 AM, Victor Tso-Guillen (vtso@paxata.com)
> wrote:
>
>  I'm thinking of local mode where multiple virtual executors occupy the
> same vm. Can we have the same configuration in spark standalone cluster
> mode?
>
>

Re: Possible to make one executor be able to work on multiple tasks simultaneously?

[Matei Zaharia <ma...@gmail.com>]

Yes, executors run one task per core of your machine by default. You can also manually launch them with more worker threads than you have cores. What cluster manager are you on?

Matei

On August 29, 2014 at 11:24:33 AM, Victor Tso-Guillen (vtso@paxata.com) wrote:

I'm thinking of local mode where multiple virtual executors occupy the same vm. Can we have the same configuration in spark standalone cluster mode?