You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Mi...@emc.com on 2011/10/11 23:05:38 UTC
Yarn NodeManager services
Hi,
As part of MPI implementation in Yarn (aka Hamster), I was looking at
refactoring some of the functionality into node manager services, so that
it can be reused by other frameworks too. (Based on the discussion with
some folks led me to believe that suffle etc is also being planned as NM
service, separating it from deep integration with node manager.)
I could not find the documentation for building NM services, in the MR-279
architecture PDF. Does the service need to reside inside the NM jvm ? How
does NM talk to the service ? How do containers discover the service ? (I
know I can look at the code, but if there is already some docs (or even
philosophical musings :-), it will make my life easier.
Thanks,
- milind
---
Milind Bhandarkar
Greenplum Labs, EMC
(Disclaimer: Opinions expressed in this email are those of the author, and
do not necessarily represent the views of any organization, past or
present, the author might be affiliated with.)
Re: Yarn NodeManager services
Posted by Mi...@emc.com.
This is cool. Exactly what I was looking for ! Thanks Chris & Mahadev.
- Milind
On 10/11/11 2:33 PM, "Chris Douglas" <cd...@apache.org> wrote:
>On Tue, Oct 11, 2011 at 2:05 PM, <Mi...@emc.com> wrote:
>> As part of MPI implementation in Yarn (aka Hamster), I was looking at
>> refactoring some of the functionality into node manager services, so
>>that
>> it can be reused by other frameworks too. (Based on the discussion with
>> some folks led me to believe that suffle etc is also being planned as NM
>> service, separating it from deep integration with node manager.)
>
>A general-purpose transfer protocol for intermediate data was designed
>and scrapped for time. So was a local service allocated and shared by
>resident containers. Instead, the NM supports loading modules (like
>ShuffleHandler) at startup that implement auxiliary (or its
>misspelling; my bad), ops-approved services, particularly shared
>protocols independent of container and application allocation.
>
>> I could not find the documentation for building NM services, in the
>>MR-279
>> architecture PDF. Does the service need to reside inside the NM jvm ?
>
>Yes.
>
>> How
>> does NM talk to the service ?
>
>The NM loads and starts a list of services from the config by
>classname. Each of these services follows the same lifecycle as the
>other NM components. Each service is given a name in the config, which
>is also its address for containers launched on that machine. Each
>container can pass an opaque blob to the service to initialize it with
>container-specific data. For the MR shuffle, these are the job
>credentials.
>
>Don't bother listing everything that's wrong with this model. Its
>principal virtue was speed of implementation.
>
>> How do containers discover the service ?
>
>They don't. They need to know the name of the service a priori. -C
>
>> (I
>> know I can look at the code, but if there is already some docs (or even
>> philosophical musings :-), it will make my life easier.
>>
>> Thanks,
>>
>> - milind
>>
>> ---
>> Milind Bhandarkar
>> Greenplum Labs, EMC
>> (Disclaimer: Opinions expressed in this email are those of the author,
>>and
>> do not necessarily represent the views of any organization, past or
>> present, the author might be affiliated with.)
>>
>>
>>
>>
>
Re: Yarn NodeManager services
Posted by Chris Douglas <cd...@apache.org>.
On Tue, Oct 11, 2011 at 2:05 PM, <Mi...@emc.com> wrote:
> As part of MPI implementation in Yarn (aka Hamster), I was looking at
> refactoring some of the functionality into node manager services, so that
> it can be reused by other frameworks too. (Based on the discussion with
> some folks led me to believe that suffle etc is also being planned as NM
> service, separating it from deep integration with node manager.)
A general-purpose transfer protocol for intermediate data was designed
and scrapped for time. So was a local service allocated and shared by
resident containers. Instead, the NM supports loading modules (like
ShuffleHandler) at startup that implement auxiliary (or its
misspelling; my bad), ops-approved services, particularly shared
protocols independent of container and application allocation.
> I could not find the documentation for building NM services, in the MR-279
> architecture PDF. Does the service need to reside inside the NM jvm ?
Yes.
> How
> does NM talk to the service ?
The NM loads and starts a list of services from the config by
classname. Each of these services follows the same lifecycle as the
other NM components. Each service is given a name in the config, which
is also its address for containers launched on that machine. Each
container can pass an opaque blob to the service to initialize it with
container-specific data. For the MR shuffle, these are the job
credentials.
Don't bother listing everything that's wrong with this model. Its
principal virtue was speed of implementation.
> How do containers discover the service ?
They don't. They need to know the name of the service a priori. -C
> (I
> know I can look at the code, but if there is already some docs (or even
> philosophical musings :-), it will make my life easier.
>
> Thanks,
>
> - milind
>
> ---
> Milind Bhandarkar
> Greenplum Labs, EMC
> (Disclaimer: Opinions expressed in this email are those of the author, and
> do not necessarily represent the views of any organization, past or
> present, the author might be affiliated with.)
>
>
>
>
Re: Yarn NodeManager services
Posted by Mahadev Konar <ma...@hortonworks.com>.
Shuffle is an auxiliary service in the NM. You might want to take a look at AuxServices.java in the NodeManager codebase.
Also, you can add config parameters to
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
(config knob in NodeManagers)
to add more configurable services in the NodeManager.
Hope that helps.
mahadev
On Oct 11, 2011, at 2:05 PM, <Mi...@emc.com> <Mi...@emc.com> wrote:
> Hi,
>
> As part of MPI implementation in Yarn (aka Hamster), I was looking at
> refactoring some of the functionality into node manager services, so that
> it can be reused by other frameworks too. (Based on the discussion with
> some folks led me to believe that suffle etc is also being planned as NM
> service, separating it from deep integration with node manager.)
>
> I could not find the documentation for building NM services, in the MR-279
> architecture PDF. Does the service need to reside inside the NM jvm ? How
> does NM talk to the service ? How do containers discover the service ? (I
> know I can look at the code, but if there is already some docs (or even
> philosophical musings :-), it will make my life easier.
>
> Thanks,
>
> - milind
>
> ---
> Milind Bhandarkar
> Greenplum Labs, EMC
> (Disclaimer: Opinions expressed in this email are those of the author, and
> do not necessarily represent the views of any organization, past or
> present, the author might be affiliated with.)
>
>
>