You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Wei Wang <xw...@gmail.com> on 2014/04/17 12:12:15 UTC
Spark on Yarn or Mesos?
Hi, there
I would like to know is there any differences between Spark on Yarn and
Spark on Mesos. Is there any comparision between them? What are the
advantages and disadvantages for each of them. Is there any criterion for
choosing between Yarn and Mesos?
BTW, we need MPI in our framework, and I saw MPICH2 is included in Mesos.
Should it be the reason for choosing Mesos?
Thanks a lot!
Weida
Re: Spark on Yarn or Mesos?
Posted by Andrew Ash <an...@andrewash.com>.
Much thanks for the perspective Matei.
On Sun, Apr 27, 2014 at 10:51 PM, Matei Zaharia <ma...@gmail.com>wrote:
> From my point of view, both are supported equally. The YARN support is
> newer and that’s why there’s been a lot more action there in recent months.
>
> Matei
>
> On Apr 27, 2014, at 12:08 PM, Andrew Ash <an...@andrewash.com> wrote:
>
> That thread was mostly about benchmarking YARN vs standalone, and the
> results are what I'd expect -- spinning up a Spark cluster on demand
> through YARN has higher startup latency than using a standalone cluster,
> where the JVMs are already initialized and ready.
>
> Given that there's a lot more commit activity around YARN as compared to
> Mesos, does that mean that YARN integration is just earlier in the maturity
> curve, or does it mean that YARN is the future and Mesos is in
> maintenance-only mode?
>
> That may be more a question for the Databricks team though: will YARN and
> Mesos be supported equally, or will one become the preferred method of
> doing cluster management under Spark?
>
> Andrew
>
>
> On Thu, Apr 17, 2014 at 1:27 PM, Arpit Tak <ar...@sigmoidanalytics.com>wrote:
>
>> Hi Wel,
>>
>> Take a look at this post...
>>
>> http://apache-spark-user-list.1001560.n3.nabble.com/Job-initialization-performance-of-Spark-standalone-mode-vs-YARN-td2016.html
>>
>> Regards,
>> Arpit Tak
>>
>>
>> On Thu, Apr 17, 2014 at 3:42 PM, Wei Wang <xw...@gmail.com> wrote:
>>
>>> Hi, there
>>>
>>> I would like to know is there any differences between Spark on Yarn and
>>> Spark on Mesos. Is there any comparision between them? What are the
>>> advantages and disadvantages for each of them. Is there any criterion for
>>> choosing between Yarn and Mesos?
>>>
>>> BTW, we need MPI in our framework, and I saw MPICH2 is included in
>>> Mesos. Should it be the reason for choosing Mesos?
>>>
>>> Thanks a lot!
>>>
>>>
>>> Weida
>>>
>>
>>
>
>
Re: Spark on Yarn or Mesos?
Posted by Matei Zaharia <ma...@gmail.com>.
From my point of view, both are supported equally. The YARN support is newer and that’s why there’s been a lot more action there in recent months.
Matei
On Apr 27, 2014, at 12:08 PM, Andrew Ash <an...@andrewash.com> wrote:
> That thread was mostly about benchmarking YARN vs standalone, and the results are what I'd expect -- spinning up a Spark cluster on demand through YARN has higher startup latency than using a standalone cluster, where the JVMs are already initialized and ready.
>
> Given that there's a lot more commit activity around YARN as compared to Mesos, does that mean that YARN integration is just earlier in the maturity curve, or does it mean that YARN is the future and Mesos is in maintenance-only mode?
>
> That may be more a question for the Databricks team though: will YARN and Mesos be supported equally, or will one become the preferred method of doing cluster management under Spark?
>
> Andrew
>
>
> On Thu, Apr 17, 2014 at 1:27 PM, Arpit Tak <ar...@sigmoidanalytics.com> wrote:
> Hi Wel,
>
> Take a look at this post...
> http://apache-spark-user-list.1001560.n3.nabble.com/Job-initialization-performance-of-Spark-standalone-mode-vs-YARN-td2016.html
>
> Regards,
> Arpit Tak
>
>
> On Thu, Apr 17, 2014 at 3:42 PM, Wei Wang <xw...@gmail.com> wrote:
> Hi, there
>
> I would like to know is there any differences between Spark on Yarn and Spark on Mesos. Is there any comparision between them? What are the advantages and disadvantages for each of them. Is there any criterion for choosing between Yarn and Mesos?
>
> BTW, we need MPI in our framework, and I saw MPICH2 is included in Mesos. Should it be the reason for choosing Mesos?
>
> Thanks a lot!
>
>
> Weida
>
>
Re: Spark on Yarn or Mesos?
Posted by Andrew Ash <an...@andrewash.com>.
That thread was mostly about benchmarking YARN vs standalone, and the
results are what I'd expect -- spinning up a Spark cluster on demand
through YARN has higher startup latency than using a standalone cluster,
where the JVMs are already initialized and ready.
Given that there's a lot more commit activity around YARN as compared to
Mesos, does that mean that YARN integration is just earlier in the maturity
curve, or does it mean that YARN is the future and Mesos is in
maintenance-only mode?
That may be more a question for the Databricks team though: will YARN and
Mesos be supported equally, or will one become the preferred method of
doing cluster management under Spark?
Andrew
On Thu, Apr 17, 2014 at 1:27 PM, Arpit Tak <ar...@sigmoidanalytics.com>wrote:
> Hi Wel,
>
> Take a look at this post...
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Job-initialization-performance-of-Spark-standalone-mode-vs-YARN-td2016.html
>
> Regards,
> Arpit Tak
>
>
> On Thu, Apr 17, 2014 at 3:42 PM, Wei Wang <xw...@gmail.com> wrote:
>
>> Hi, there
>>
>> I would like to know is there any differences between Spark on Yarn and
>> Spark on Mesos. Is there any comparision between them? What are the
>> advantages and disadvantages for each of them. Is there any criterion for
>> choosing between Yarn and Mesos?
>>
>> BTW, we need MPI in our framework, and I saw MPICH2 is included in Mesos.
>> Should it be the reason for choosing Mesos?
>>
>> Thanks a lot!
>>
>>
>> Weida
>>
>
>
Re: Spark on Yarn or Mesos?
Posted by Arpit Tak <ar...@sigmoidanalytics.com>.
Hi Wel,
Take a look at this post...
http://apache-spark-user-list.1001560.n3.nabble.com/Job-initialization-performance-of-Spark-standalone-mode-vs-YARN-td2016.html
Regards,
Arpit Tak
On Thu, Apr 17, 2014 at 3:42 PM, Wei Wang <xw...@gmail.com> wrote:
> Hi, there
>
> I would like to know is there any differences between Spark on Yarn and
> Spark on Mesos. Is there any comparision between them? What are the
> advantages and disadvantages for each of them. Is there any criterion for
> choosing between Yarn and Mesos?
>
> BTW, we need MPI in our framework, and I saw MPICH2 is included in Mesos.
> Should it be the reason for choosing Mesos?
>
> Thanks a lot!
>
>
> Weida
>