Re: AM creation in yarn-client mode

[praveen S <my...@gmail.com>]

Can you explain what happens in yarn client mode?

Regards,
Praveen
On 10 Feb 2016 10:55, "ayan guha" <gu...@gmail.com> wrote:

> It depends on yarn-cluster and yarn-client mode.
>
> On Wed, Feb 10, 2016 at 3:42 PM, praveen S <my...@gmail.com> wrote:
>
>> Hi,
>>
>> I have 2 questions when running the spark jobs on yarn in client mode :
>>
>> 1) Where is the AM(application master) created :
>>
>> A) is it created on the client where the job was submitted? i.e driver
>> and AM on the same client?
>> Or
>> B) yarn decides where the the AM should be created?
>>
>> 2) Driver and AM run in different processes : is my assumption correct?
>>
>> Regards,
>> Praveen
>>
>
>
>
> --
> Best Regards,
> Ayan Guha
>

Re: AM creation in yarn-client mode

[Alexander Pivovarov <ap...@gmail.com>]

the pictures to illustrate it
http://www.cloudera.com/documentation/enterprise/5-4-x/topics/cdh_ig_running_spark_on_yarn.html

On Tue, Feb 9, 2016 at 10:18 PM, Jonathan Kelly <jo...@gmail.com>
wrote:

> In yarn-client mode, the driver is separate from the AM. The AM is created
> in YARN, and YARN controls where it goes (though you can somewhat control
> it using YARN node labels--I just learned earlier today in a different
> thread on this list that this can be controlled by
> spark.yarn.am.labelExpression). Then what I understand is that the driver
> talks to the AM in order to request additional YARN containers in which to
> run executors.
>
> In yarn-cluster mode, the SparkSubmit process outside of the cluster
> creates the AM in YARN, and then what I understand is that the AM *becomes*
> the driver (by invoking the driver's main method), and then it requests the
> executor containers.
>
> So yes, one difference between yarn-client and yarn-cluster mode is that
> in yarn-client mode the driver and AM are separate, whereas they are the
> same in yarn-cluster.
>
> ~ Jonathan
>
> On Tue, Feb 9, 2016 at 9:57 PM praveen S <my...@gmail.com> wrote:
>
>> Can you explain what happens in yarn client mode?
>>
>> Regards,
>> Praveen
>> On 10 Feb 2016 10:55, "ayan guha" <gu...@gmail.com> wrote:
>>
>>> It depends on yarn-cluster and yarn-client mode.
>>>
>>> On Wed, Feb 10, 2016 at 3:42 PM, praveen S <my...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have 2 questions when running the spark jobs on yarn in client mode :
>>>>
>>>> 1) Where is the AM(application master) created :
>>>>
>>>> A) is it created on the client where the job was submitted? i.e driver
>>>> and AM on the same client?
>>>> Or
>>>> B) yarn decides where the the AM should be created?
>>>>
>>>> 2) Driver and AM run in different processes : is my assumption correct?
>>>>
>>>> Regards,
>>>> Praveen
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Ayan Guha
>>>
>>

Re: AM creation in yarn-client mode

[Jonathan Kelly <jo...@gmail.com>]

In yarn-client mode, the driver is separate from the AM. The AM is created
in YARN, and YARN controls where it goes (though you can somewhat control
it using YARN node labels--I just learned earlier today in a different
thread on this list that this can be controlled by
spark.yarn.am.labelExpression). Then what I understand is that the driver
talks to the AM in order to request additional YARN containers in which to
run executors.

In yarn-cluster mode, the SparkSubmit process outside of the cluster
creates the AM in YARN, and then what I understand is that the AM *becomes*
the driver (by invoking the driver's main method), and then it requests the
executor containers.

So yes, one difference between yarn-client and yarn-cluster mode is that in
yarn-client mode the driver and AM are separate, whereas they are the same
in yarn-cluster.

~ Jonathan
On Tue, Feb 9, 2016 at 9:57 PM praveen S <my...@gmail.com> wrote:

> Can you explain what happens in yarn client mode?
>
> Regards,
> Praveen
> On 10 Feb 2016 10:55, "ayan guha" <gu...@gmail.com> wrote:
>
>> It depends on yarn-cluster and yarn-client mode.
>>
>> On Wed, Feb 10, 2016 at 3:42 PM, praveen S <my...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have 2 questions when running the spark jobs on yarn in client mode :
>>>
>>> 1) Where is the AM(application master) created :
>>>
>>> A) is it created on the client where the job was submitted? i.e driver
>>> and AM on the same client?
>>> Or
>>> B) yarn decides where the the AM should be created?
>>>
>>> 2) Driver and AM run in different processes : is my assumption correct?
>>>
>>> Regards,
>>> Praveen
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Ayan Guha
>>
>