You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Ruslan Dautkhanov <da...@gmail.com> on 2015/08/16 22:59:48 UTC

Re: Spark Master HA on YARN

There is no Spark master in YARN mode. It's standalone mode terminology.
In YARN cluster mode, Spark's Application Master (Spark Driver runs in it)
will be restarted
automatically by RM up to yarn.resourcemanager.am.max-retries
times (default is 2).

--
Ruslan Dautkhanov

On Fri, Jul 17, 2015 at 1:29 AM, Bhaskar Dutta <bh...@gmail.com> wrote:

> Hi,
>
> Is Spark master high availability supported on YARN (yarn-client mode)
> analogous to
> https://spark.apache.org/docs/1.4.0/spark-standalone.html#high-availability
> ?
>
> Thanks
> Bhaskie
>

Re: Spark Master HA on YARN

Posted by Jeff Zhang <zj...@gmail.com>.

To make it clear,  Spark Standalone is similar to Yarn as a simple cluster
management system.

Spark Master  <--->   Yarn Resource Manager
Spark Worker  <--->   Yarn Node Manager

On Mon, Aug 17, 2015 at 4:59 AM, Ruslan Dautkhanov <da...@gmail.com>
wrote:

> There is no Spark master in YARN mode. It's standalone mode terminology.
> In YARN cluster mode, Spark's Application Master (Spark Driver runs in it)
> will be restarted
> automatically by RM up to yarn.resourcemanager.am.max-retries
> times (default is 2).
>
> --
> Ruslan Dautkhanov
>
> On Fri, Jul 17, 2015 at 1:29 AM, Bhaskar Dutta <bh...@gmail.com> wrote:
>
>> Hi,
>>
>> Is Spark master high availability supported on YARN (yarn-client mode)
>> analogous to
>> https://spark.apache.org/docs/1.4.0/spark-standalone.html#high-availability
>> ?
>>
>> Thanks
>> Bhaskie
>>
>
>


-- 
Best Regards

Jeff Zhang