You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by on 2015/08/12 23:16:30 UTC
Spark - Standalone Vs YARN Vs Mesos
Do we have any comparisons in terms of resource utilization, scheduling of
running Spark in the below three modes
1) Standalone
2) over YARN
3) over Mesos
Can some one share resources (thoughts/URLs) on this area.
--
Deepak
Re: Spark - Standalone Vs YARN Vs Mesos
Posted by .
What are ideas around Spark cluster for streaming purposes ?
What is better standalone / Mesos / YARN ?
Please share cluster details and size of data and type of processing.
(multiple processing points) (architecture or similar)
I see folks using YARN cluster for streaming purposes.
Regards,
Deepak
On Thu, Aug 13, 2015 at 9:12 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
> I am looking to decide what is best for my production grade spark
> application(s).
>
> YARN
> =====
>
> 1. YARN supports security. When Spark is run over YARN the
> communication between processes can use secure authentication through
> Kerberos.
> 2. Spark standalone cluster can only run Spark jobs and nothing else.
> With YARN you can have different kinds of jobs like M/R, or Spark.
> 3. YARN scheduler has multiple features like Queues, hierarchical
> queues with pluggable policies, auto placement of apps into queues, easy
> installation, ACLs for queues. These are missing with standalone spark
> scheduler. Resources are more intelligently and dynamically used.
> 4. Spark standalone scheduler requires each application to run an
> executor on every node in cluster, whereas with YARN you can run
> executor(s) on subset of nodes. I haven't tested it.
> 5. On YARN spark supports driver to run on the client machine itself
> (yarn-client) which requires the client application to run for the lifetime
> of application. With yarn-cluster mode, the spark driver will run on
> Application master and hence the client program can either exit or do
> something else. This feature might be missing with standalone scheduler.
> 6. YARN provides finer control of resources like CPU cores. Number of
> executors per node is configurable with YARN depending on the number of
> CPUs present on the node, this is missing on Mesos. It might be
> available with future releases.
> 7. Standalone mode requires management of daemon services. It also
> requires a Zookeeper setup as Spark master node needs to be highly
> available to avoid single point of failure.
> 8. Most of the existing users of Hadoop cluster have large amounts
> data (TBs/PBs) on residing on the cluster and hence Spark applications can
> make use of data locality when running on YARN cluster. Making it available
> on standalone cluster might be a challenge.
> 9. I do not think there are performance impacts of running a spark
> application on YARN/Mesos/Standalone cluster. Might require a test.
> 10. Mesos & Spark both were developed at Amplab, so both might be
> better compatible with each other. However I do not have any working
> knowledge of Mesos.
>
>
>
> I was thinking what are advantages and disadvantages of running Spark over
> Mesos and Spark over Standalone cluster. This will help me ( and others on
> the verge of using Spark systems) to decide which direction to go.
>
> Regards,
> Deepak
>
> On Wed, 12 Aug 2015 at 10:28 PM Tim Chen <ti...@mesosphere.io> wrote:
>
>> I'm not sure what you're looking for, since you can't really compare
>> Standalone with YARN or Mesos, as Standalone is assuming the Spark
>> workers/master owns the cluster, and YARN/Mesos is trying to share the
>> cluster among different applications/frameworks.
>>
>> And when you refer to resource utilization, what exactly does it mean to
>> you? Is it the ability to maximize the usage of your resources with
>> multiple applications in mind, or just how much configuration Spark allows
>> you to in each mode?
>>
>> Tim
>>
>> On Wed, Aug 12, 2015 at 2:16 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>> wrote:
>>
>>> Do we have any comparisons in terms of resource utilization, scheduling
>>> of running Spark in the below three modes
>>> 1) Standalone
>>> 2) over YARN
>>> 3) over Mesos
>>>
>>> Can some one share resources (thoughts/URLs) on this area.
>>>
>>>
>>> --
>>> Deepak
>>>
>>>
>>
--
Deepak
Re: Spark - Standalone Vs YARN Vs Mesos
Posted by .
I am looking to decide what is best for my production grade spark
application(s).
YARN
=====
1. YARN supports security. When Spark is run over YARN the communication
between processes can use secure authentication through Kerberos.
2. Spark standalone cluster can only run Spark jobs and nothing else.
With YARN you can have different kinds of jobs like M/R, or Spark.
3. YARN scheduler has multiple features like Queues, hierarchical queues
with pluggable policies, auto placement of apps into queues, easy
installation, ACLs for queues. These are missing with standalone spark
scheduler. Resources are more intelligently and dynamically used.
4. Spark standalone scheduler requires each application to run an
executor on every node in cluster, whereas with YARN you can run
executor(s) on subset of nodes. I haven't tested it.
5. On YARN spark supports driver to run on the client machine itself
(yarn-client) which requires the client application to run for the lifetime
of application. With yarn-cluster mode, the spark driver will run on
Application master and hence the client program can either exit or do
something else. This feature might be missing with standalone scheduler.
6. YARN provides finer control of resources like CPU cores. Number of
executors per node is configurable with YARN depending on the number of
CPUs present on the node, this is missing on Mesos. It might be
available with future releases.
7. Standalone mode requires management of daemon services. It also
requires a Zookeeper setup as Spark master node needs to be highly
available to avoid single point of failure.
8. Most of the existing users of Hadoop cluster have large amounts data
(TBs/PBs) on residing on the cluster and hence Spark applications can make
use of data locality when running on YARN cluster. Making it available on
standalone cluster might be a challenge.
9. I do not think there are performance impacts of running a spark
application on YARN/Mesos/Standalone cluster. Might require a test.
10. Mesos & Spark both were developed at Amplab, so both might be better
compatible with each other. However I do not have any working knowledge of
Mesos.
I was thinking what are advantages and disadvantages of running Spark over
Mesos and Spark over Standalone cluster. This will help me ( and others on
the verge of using Spark systems) to decide which direction to go.
Regards,
Deepak
On Wed, 12 Aug 2015 at 10:28 PM Tim Chen <ti...@mesosphere.io> wrote:
> I'm not sure what you're looking for, since you can't really compare
> Standalone with YARN or Mesos, as Standalone is assuming the Spark
> workers/master owns the cluster, and YARN/Mesos is trying to share the
> cluster among different applications/frameworks.
>
> And when you refer to resource utilization, what exactly does it mean to
> you? Is it the ability to maximize the usage of your resources with
> multiple applications in mind, or just how much configuration Spark allows
> you to in each mode?
>
> Tim
>
> On Wed, Aug 12, 2015 at 2:16 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
> wrote:
>
>> Do we have any comparisons in terms of resource utilization, scheduling
>> of running Spark in the below three modes
>> 1) Standalone
>> 2) over YARN
>> 3) over Mesos
>>
>> Can some one share resources (thoughts/URLs) on this area.
>>
>>
>> --
>> Deepak
>>
>>
>
Re: Spark - Standalone Vs YARN Vs Mesos
Posted by Tim Chen <ti...@mesosphere.io>.
I'm not sure what you're looking for, since you can't really compare
Standalone with YARN or Mesos, as Standalone is assuming the Spark
workers/master owns the cluster, and YARN/Mesos is trying to share the
cluster among different applications/frameworks.
And when you refer to resource utilization, what exactly does it mean to
you? Is it the ability to maximize the usage of your resources with
multiple applications in mind, or just how much configuration Spark allows
you to in each mode?
Tim
On Wed, Aug 12, 2015 at 2:16 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
> Do we have any comparisons in terms of resource utilization, scheduling of
> running Spark in the below three modes
> 1) Standalone
> 2) over YARN
> 3) over Mesos
>
> Can some one share resources (thoughts/URLs) on this area.
>
>
> --
> Deepak
>
>