You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Dongwon Kim <ea...@postech.ac.kr> on 2015/04/01 08:38:32 UTC
Re: Run my own application master on a specific node in a YARN cluster
Thanks for your input but I need to launch my own node manager
(different from the Yarn NM) running on each node.
(which is not explained in the original question)
If I were to launch just a single master with a well-known address,
ZooKeeper would be a great solution!
Thanks.
Dongwon Kim
2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> Hi,
>
> In these circumstances, there is no easy way to do that. Maybe use
> workaround. How about using zookeeper for shared storage? The app master
> create predefined zookeeper node when starting with current machine's IP and
> Clients always look for that zookeeper node for app master's location.
>
> Thanks.
>
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim <ea...@postech.ac.kr>
> wrote:
>>
>> Hello,
>>
>> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
>> master on a specific node in a YARN cluster in order to open a server
>> on a predetermined IP address and port. To that end, I wrote a driver
>> program in which I created a ResourceRequest object and called
>> setResourceName method to set a hostname, and attached it to a
>> ApplicationSubmissionContext object by
>> callingsetAMContainerResourceRequest method.
>>
>> I tried several times but couldn't launch the app master on a specific
>> node. After searching code, I found that RMAppAttemptImpl invalidates
>> what I've set in ResourceRequest as follows:
>>
>> // Currently, following fields are all hard code,
>> // TODO: change these fields when we want to support
>> // priority/resource-name/relax-locality specification for AM
>> containers
>> // allocation.
>> appAttempt.amReq.setNumContainers(1);
>> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
>> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
>> appAttempt.amReq.setRelaxLocality(true);
>>
>> Is there another way to launch a container for an application master
>> on a specific node in Hadoop-2.6.0?
>>
>> Thanks.
>>
>> Dongwon Kim
>
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Dongwon Kim <ea...@postech.ac.kr>.
I'm doing precisely the opposite.
My own node manager (MY_NM) is an AM in YARN and, therefore, each
MY_NM is expected to run inside a YARN container.
What I trying to do is to execute the AM (MY_NM) on each slave.
For the reason, I need to launch an AM on a specific node but
Hadoop-2.6.0 ignores what I describe in a ResourceRequest object.
According to what I've understood, RMAppManager in ResourceManager
creates a RMAppImpl object for each application and a RMAppAttempt
object for each application attempt.
When the state of RMAppAttempt changes from SUBMITTED to SCHEDULED,
RMAppAttempt invalidates necessary information to launch a container
for AM such as # containers, priority, resource name (=hostname),
whether to relax locality as in the above code and then asks
YarnScheduler to allocate a container.
I conclude that any application master cannot be launched on a
specific node in Hadoop-2.6.0.
As a workaround, I try to execute an AM directly on a specific node
from the command line without submitting it to YARN and register the
AM to RM.
It however fails (of course :P) because the AM is not given a
delegation token necessary to communicate with a RPC server in RM.
Anyway I need to find another way of doing that or I have to alter the
design of my framework.
Thanks.
- Dongwon Kim
2015-04-01 17:50 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> Very interesting, BTW. So you try to launch app-master with YARN Container
> but your own node-manager without YARN Container, Am I right?
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Wed, Apr 1, 2015 at 3:38 PM, Dongwon Kim <ea...@postech.ac.kr>
> wrote:
>>
>> Thanks for your input but I need to launch my own node manager
>> (different from the Yarn NM) running on each node.
>> (which is not explained in the original question)
>>
>> If I were to launch just a single master with a well-known address,
>> ZooKeeper would be a great solution!
>> Thanks.
>>
>> Dongwon Kim
>>
>> 2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
>> > Hi,
>> >
>> > In these circumstances, there is no easy way to do that. Maybe use
>> > workaround. How about using zookeeper for shared storage? The app master
>> > create predefined zookeeper node when starting with current machine's IP
>> > and
>> > Clients always look for that zookeeper node for app master's location.
>> >
>> > Thanks.
>> >
>> >
>> > Drake 민영근 Ph.D
>> > kt NexR
>> >
>> > On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim
>> > <ea...@postech.ac.kr>
>> > wrote:
>> >>
>> >> Hello,
>> >>
>> >> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
>> >> master on a specific node in a YARN cluster in order to open a server
>> >> on a predetermined IP address and port. To that end, I wrote a driver
>> >> program in which I created a ResourceRequest object and called
>> >> setResourceName method to set a hostname, and attached it to a
>> >> ApplicationSubmissionContext object by
>> >> callingsetAMContainerResourceRequest method.
>> >>
>> >> I tried several times but couldn't launch the app master on a specific
>> >> node. After searching code, I found that RMAppAttemptImpl invalidates
>> >> what I've set in ResourceRequest as follows:
>> >>
>> >> // Currently, following fields are all hard code,
>> >> // TODO: change these fields when we want to support
>> >> // priority/resource-name/relax-locality specification for AM
>> >> containers
>> >> // allocation.
>> >> appAttempt.amReq.setNumContainers(1);
>> >> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
>> >> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
>> >> appAttempt.amReq.setRelaxLocality(true);
>> >>
>> >> Is there another way to launch a container for an application master
>> >> on a specific node in Hadoop-2.6.0?
>> >>
>> >> Thanks.
>> >>
>> >> Dongwon Kim
>> >
>> >
>
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Dongwon Kim <ea...@postech.ac.kr>.
I'm doing precisely the opposite.
My own node manager (MY_NM) is an AM in YARN and, therefore, each
MY_NM is expected to run inside a YARN container.
What I trying to do is to execute the AM (MY_NM) on each slave.
For the reason, I need to launch an AM on a specific node but
Hadoop-2.6.0 ignores what I describe in a ResourceRequest object.
According to what I've understood, RMAppManager in ResourceManager
creates a RMAppImpl object for each application and a RMAppAttempt
object for each application attempt.
When the state of RMAppAttempt changes from SUBMITTED to SCHEDULED,
RMAppAttempt invalidates necessary information to launch a container
for AM such as # containers, priority, resource name (=hostname),
whether to relax locality as in the above code and then asks
YarnScheduler to allocate a container.
I conclude that any application master cannot be launched on a
specific node in Hadoop-2.6.0.
As a workaround, I try to execute an AM directly on a specific node
from the command line without submitting it to YARN and register the
AM to RM.
It however fails (of course :P) because the AM is not given a
delegation token necessary to communicate with a RPC server in RM.
Anyway I need to find another way of doing that or I have to alter the
design of my framework.
Thanks.
- Dongwon Kim
2015-04-01 17:50 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> Very interesting, BTW. So you try to launch app-master with YARN Container
> but your own node-manager without YARN Container, Am I right?
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Wed, Apr 1, 2015 at 3:38 PM, Dongwon Kim <ea...@postech.ac.kr>
> wrote:
>>
>> Thanks for your input but I need to launch my own node manager
>> (different from the Yarn NM) running on each node.
>> (which is not explained in the original question)
>>
>> If I were to launch just a single master with a well-known address,
>> ZooKeeper would be a great solution!
>> Thanks.
>>
>> Dongwon Kim
>>
>> 2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
>> > Hi,
>> >
>> > In these circumstances, there is no easy way to do that. Maybe use
>> > workaround. How about using zookeeper for shared storage? The app master
>> > create predefined zookeeper node when starting with current machine's IP
>> > and
>> > Clients always look for that zookeeper node for app master's location.
>> >
>> > Thanks.
>> >
>> >
>> > Drake 민영근 Ph.D
>> > kt NexR
>> >
>> > On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim
>> > <ea...@postech.ac.kr>
>> > wrote:
>> >>
>> >> Hello,
>> >>
>> >> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
>> >> master on a specific node in a YARN cluster in order to open a server
>> >> on a predetermined IP address and port. To that end, I wrote a driver
>> >> program in which I created a ResourceRequest object and called
>> >> setResourceName method to set a hostname, and attached it to a
>> >> ApplicationSubmissionContext object by
>> >> callingsetAMContainerResourceRequest method.
>> >>
>> >> I tried several times but couldn't launch the app master on a specific
>> >> node. After searching code, I found that RMAppAttemptImpl invalidates
>> >> what I've set in ResourceRequest as follows:
>> >>
>> >> // Currently, following fields are all hard code,
>> >> // TODO: change these fields when we want to support
>> >> // priority/resource-name/relax-locality specification for AM
>> >> containers
>> >> // allocation.
>> >> appAttempt.amReq.setNumContainers(1);
>> >> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
>> >> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
>> >> appAttempt.amReq.setRelaxLocality(true);
>> >>
>> >> Is there another way to launch a container for an application master
>> >> on a specific node in Hadoop-2.6.0?
>> >>
>> >> Thanks.
>> >>
>> >> Dongwon Kim
>> >
>> >
>
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Dongwon Kim <ea...@postech.ac.kr>.
I'm doing precisely the opposite.
My own node manager (MY_NM) is an AM in YARN and, therefore, each
MY_NM is expected to run inside a YARN container.
What I trying to do is to execute the AM (MY_NM) on each slave.
For the reason, I need to launch an AM on a specific node but
Hadoop-2.6.0 ignores what I describe in a ResourceRequest object.
According to what I've understood, RMAppManager in ResourceManager
creates a RMAppImpl object for each application and a RMAppAttempt
object for each application attempt.
When the state of RMAppAttempt changes from SUBMITTED to SCHEDULED,
RMAppAttempt invalidates necessary information to launch a container
for AM such as # containers, priority, resource name (=hostname),
whether to relax locality as in the above code and then asks
YarnScheduler to allocate a container.
I conclude that any application master cannot be launched on a
specific node in Hadoop-2.6.0.
As a workaround, I try to execute an AM directly on a specific node
from the command line without submitting it to YARN and register the
AM to RM.
It however fails (of course :P) because the AM is not given a
delegation token necessary to communicate with a RPC server in RM.
Anyway I need to find another way of doing that or I have to alter the
design of my framework.
Thanks.
- Dongwon Kim
2015-04-01 17:50 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> Very interesting, BTW. So you try to launch app-master with YARN Container
> but your own node-manager without YARN Container, Am I right?
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Wed, Apr 1, 2015 at 3:38 PM, Dongwon Kim <ea...@postech.ac.kr>
> wrote:
>>
>> Thanks for your input but I need to launch my own node manager
>> (different from the Yarn NM) running on each node.
>> (which is not explained in the original question)
>>
>> If I were to launch just a single master with a well-known address,
>> ZooKeeper would be a great solution!
>> Thanks.
>>
>> Dongwon Kim
>>
>> 2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
>> > Hi,
>> >
>> > In these circumstances, there is no easy way to do that. Maybe use
>> > workaround. How about using zookeeper for shared storage? The app master
>> > create predefined zookeeper node when starting with current machine's IP
>> > and
>> > Clients always look for that zookeeper node for app master's location.
>> >
>> > Thanks.
>> >
>> >
>> > Drake 민영근 Ph.D
>> > kt NexR
>> >
>> > On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim
>> > <ea...@postech.ac.kr>
>> > wrote:
>> >>
>> >> Hello,
>> >>
>> >> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
>> >> master on a specific node in a YARN cluster in order to open a server
>> >> on a predetermined IP address and port. To that end, I wrote a driver
>> >> program in which I created a ResourceRequest object and called
>> >> setResourceName method to set a hostname, and attached it to a
>> >> ApplicationSubmissionContext object by
>> >> callingsetAMContainerResourceRequest method.
>> >>
>> >> I tried several times but couldn't launch the app master on a specific
>> >> node. After searching code, I found that RMAppAttemptImpl invalidates
>> >> what I've set in ResourceRequest as follows:
>> >>
>> >> // Currently, following fields are all hard code,
>> >> // TODO: change these fields when we want to support
>> >> // priority/resource-name/relax-locality specification for AM
>> >> containers
>> >> // allocation.
>> >> appAttempt.amReq.setNumContainers(1);
>> >> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
>> >> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
>> >> appAttempt.amReq.setRelaxLocality(true);
>> >>
>> >> Is there another way to launch a container for an application master
>> >> on a specific node in Hadoop-2.6.0?
>> >>
>> >> Thanks.
>> >>
>> >> Dongwon Kim
>> >
>> >
>
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Dongwon Kim <ea...@postech.ac.kr>.
I'm doing precisely the opposite.
My own node manager (MY_NM) is an AM in YARN and, therefore, each
MY_NM is expected to run inside a YARN container.
What I trying to do is to execute the AM (MY_NM) on each slave.
For the reason, I need to launch an AM on a specific node but
Hadoop-2.6.0 ignores what I describe in a ResourceRequest object.
According to what I've understood, RMAppManager in ResourceManager
creates a RMAppImpl object for each application and a RMAppAttempt
object for each application attempt.
When the state of RMAppAttempt changes from SUBMITTED to SCHEDULED,
RMAppAttempt invalidates necessary information to launch a container
for AM such as # containers, priority, resource name (=hostname),
whether to relax locality as in the above code and then asks
YarnScheduler to allocate a container.
I conclude that any application master cannot be launched on a
specific node in Hadoop-2.6.0.
As a workaround, I try to execute an AM directly on a specific node
from the command line without submitting it to YARN and register the
AM to RM.
It however fails (of course :P) because the AM is not given a
delegation token necessary to communicate with a RPC server in RM.
Anyway I need to find another way of doing that or I have to alter the
design of my framework.
Thanks.
- Dongwon Kim
2015-04-01 17:50 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> Very interesting, BTW. So you try to launch app-master with YARN Container
> but your own node-manager without YARN Container, Am I right?
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Wed, Apr 1, 2015 at 3:38 PM, Dongwon Kim <ea...@postech.ac.kr>
> wrote:
>>
>> Thanks for your input but I need to launch my own node manager
>> (different from the Yarn NM) running on each node.
>> (which is not explained in the original question)
>>
>> If I were to launch just a single master with a well-known address,
>> ZooKeeper would be a great solution!
>> Thanks.
>>
>> Dongwon Kim
>>
>> 2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
>> > Hi,
>> >
>> > In these circumstances, there is no easy way to do that. Maybe use
>> > workaround. How about using zookeeper for shared storage? The app master
>> > create predefined zookeeper node when starting with current machine's IP
>> > and
>> > Clients always look for that zookeeper node for app master's location.
>> >
>> > Thanks.
>> >
>> >
>> > Drake 민영근 Ph.D
>> > kt NexR
>> >
>> > On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim
>> > <ea...@postech.ac.kr>
>> > wrote:
>> >>
>> >> Hello,
>> >>
>> >> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
>> >> master on a specific node in a YARN cluster in order to open a server
>> >> on a predetermined IP address and port. To that end, I wrote a driver
>> >> program in which I created a ResourceRequest object and called
>> >> setResourceName method to set a hostname, and attached it to a
>> >> ApplicationSubmissionContext object by
>> >> callingsetAMContainerResourceRequest method.
>> >>
>> >> I tried several times but couldn't launch the app master on a specific
>> >> node. After searching code, I found that RMAppAttemptImpl invalidates
>> >> what I've set in ResourceRequest as follows:
>> >>
>> >> // Currently, following fields are all hard code,
>> >> // TODO: change these fields when we want to support
>> >> // priority/resource-name/relax-locality specification for AM
>> >> containers
>> >> // allocation.
>> >> appAttempt.amReq.setNumContainers(1);
>> >> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
>> >> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
>> >> appAttempt.amReq.setRelaxLocality(true);
>> >>
>> >> Is there another way to launch a container for an application master
>> >> on a specific node in Hadoop-2.6.0?
>> >>
>> >> Thanks.
>> >>
>> >> Dongwon Kim
>> >
>> >
>
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Drake민영근 <dr...@nexr.com>.
Very interesting, BTW. So you try to launch app-master with YARN Container
but your own node-manager without YARN Container, Am I right?
Drake 민영근 Ph.D
kt NexR
On Wed, Apr 1, 2015 at 3:38 PM, Dongwon Kim <ea...@postech.ac.kr>
wrote:
> Thanks for your input but I need to launch my own node manager
> (different from the Yarn NM) running on each node.
> (which is not explained in the original question)
>
> If I were to launch just a single master with a well-known address,
> ZooKeeper would be a great solution!
> Thanks.
>
> Dongwon Kim
>
> 2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> > Hi,
> >
> > In these circumstances, there is no easy way to do that. Maybe use
> > workaround. How about using zookeeper for shared storage? The app master
> > create predefined zookeeper node when starting with current machine's IP
> and
> > Clients always look for that zookeeper node for app master's location.
> >
> > Thanks.
> >
> >
> > Drake 민영근 Ph.D
> > kt NexR
> >
> > On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim <eastcirclek@postech.ac.kr
> >
> > wrote:
> >>
> >> Hello,
> >>
> >> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
> >> master on a specific node in a YARN cluster in order to open a server
> >> on a predetermined IP address and port. To that end, I wrote a driver
> >> program in which I created a ResourceRequest object and called
> >> setResourceName method to set a hostname, and attached it to a
> >> ApplicationSubmissionContext object by
> >> callingsetAMContainerResourceRequest method.
> >>
> >> I tried several times but couldn't launch the app master on a specific
> >> node. After searching code, I found that RMAppAttemptImpl invalidates
> >> what I've set in ResourceRequest as follows:
> >>
> >> // Currently, following fields are all hard code,
> >> // TODO: change these fields when we want to support
> >> // priority/resource-name/relax-locality specification for AM
> >> containers
> >> // allocation.
> >> appAttempt.amReq.setNumContainers(1);
> >> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
> >> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
> >> appAttempt.amReq.setRelaxLocality(true);
> >>
> >> Is there another way to launch a container for an application master
> >> on a specific node in Hadoop-2.6.0?
> >>
> >> Thanks.
> >>
> >> Dongwon Kim
> >
> >
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Drake민영근 <dr...@nexr.com>.
Very interesting, BTW. So you try to launch app-master with YARN Container
but your own node-manager without YARN Container, Am I right?
Drake 민영근 Ph.D
kt NexR
On Wed, Apr 1, 2015 at 3:38 PM, Dongwon Kim <ea...@postech.ac.kr>
wrote:
> Thanks for your input but I need to launch my own node manager
> (different from the Yarn NM) running on each node.
> (which is not explained in the original question)
>
> If I were to launch just a single master with a well-known address,
> ZooKeeper would be a great solution!
> Thanks.
>
> Dongwon Kim
>
> 2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> > Hi,
> >
> > In these circumstances, there is no easy way to do that. Maybe use
> > workaround. How about using zookeeper for shared storage? The app master
> > create predefined zookeeper node when starting with current machine's IP
> and
> > Clients always look for that zookeeper node for app master's location.
> >
> > Thanks.
> >
> >
> > Drake 민영근 Ph.D
> > kt NexR
> >
> > On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim <eastcirclek@postech.ac.kr
> >
> > wrote:
> >>
> >> Hello,
> >>
> >> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
> >> master on a specific node in a YARN cluster in order to open a server
> >> on a predetermined IP address and port. To that end, I wrote a driver
> >> program in which I created a ResourceRequest object and called
> >> setResourceName method to set a hostname, and attached it to a
> >> ApplicationSubmissionContext object by
> >> callingsetAMContainerResourceRequest method.
> >>
> >> I tried several times but couldn't launch the app master on a specific
> >> node. After searching code, I found that RMAppAttemptImpl invalidates
> >> what I've set in ResourceRequest as follows:
> >>
> >> // Currently, following fields are all hard code,
> >> // TODO: change these fields when we want to support
> >> // priority/resource-name/relax-locality specification for AM
> >> containers
> >> // allocation.
> >> appAttempt.amReq.setNumContainers(1);
> >> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
> >> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
> >> appAttempt.amReq.setRelaxLocality(true);
> >>
> >> Is there another way to launch a container for an application master
> >> on a specific node in Hadoop-2.6.0?
> >>
> >> Thanks.
> >>
> >> Dongwon Kim
> >
> >
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Drake민영근 <dr...@nexr.com>.
Very interesting, BTW. So you try to launch app-master with YARN Container
but your own node-manager without YARN Container, Am I right?
Drake 민영근 Ph.D
kt NexR
On Wed, Apr 1, 2015 at 3:38 PM, Dongwon Kim <ea...@postech.ac.kr>
wrote:
> Thanks for your input but I need to launch my own node manager
> (different from the Yarn NM) running on each node.
> (which is not explained in the original question)
>
> If I were to launch just a single master with a well-known address,
> ZooKeeper would be a great solution!
> Thanks.
>
> Dongwon Kim
>
> 2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> > Hi,
> >
> > In these circumstances, there is no easy way to do that. Maybe use
> > workaround. How about using zookeeper for shared storage? The app master
> > create predefined zookeeper node when starting with current machine's IP
> and
> > Clients always look for that zookeeper node for app master's location.
> >
> > Thanks.
> >
> >
> > Drake 민영근 Ph.D
> > kt NexR
> >
> > On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim <eastcirclek@postech.ac.kr
> >
> > wrote:
> >>
> >> Hello,
> >>
> >> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
> >> master on a specific node in a YARN cluster in order to open a server
> >> on a predetermined IP address and port. To that end, I wrote a driver
> >> program in which I created a ResourceRequest object and called
> >> setResourceName method to set a hostname, and attached it to a
> >> ApplicationSubmissionContext object by
> >> callingsetAMContainerResourceRequest method.
> >>
> >> I tried several times but couldn't launch the app master on a specific
> >> node. After searching code, I found that RMAppAttemptImpl invalidates
> >> what I've set in ResourceRequest as follows:
> >>
> >> // Currently, following fields are all hard code,
> >> // TODO: change these fields when we want to support
> >> // priority/resource-name/relax-locality specification for AM
> >> containers
> >> // allocation.
> >> appAttempt.amReq.setNumContainers(1);
> >> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
> >> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
> >> appAttempt.amReq.setRelaxLocality(true);
> >>
> >> Is there another way to launch a container for an application master
> >> on a specific node in Hadoop-2.6.0?
> >>
> >> Thanks.
> >>
> >> Dongwon Kim
> >
> >
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Drake민영근 <dr...@nexr.com>.
Very interesting, BTW. So you try to launch app-master with YARN Container
but your own node-manager without YARN Container, Am I right?
Drake 민영근 Ph.D
kt NexR
On Wed, Apr 1, 2015 at 3:38 PM, Dongwon Kim <ea...@postech.ac.kr>
wrote:
> Thanks for your input but I need to launch my own node manager
> (different from the Yarn NM) running on each node.
> (which is not explained in the original question)
>
> If I were to launch just a single master with a well-known address,
> ZooKeeper would be a great solution!
> Thanks.
>
> Dongwon Kim
>
> 2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> > Hi,
> >
> > In these circumstances, there is no easy way to do that. Maybe use
> > workaround. How about using zookeeper for shared storage? The app master
> > create predefined zookeeper node when starting with current machine's IP
> and
> > Clients always look for that zookeeper node for app master's location.
> >
> > Thanks.
> >
> >
> > Drake 민영근 Ph.D
> > kt NexR
> >
> > On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim <eastcirclek@postech.ac.kr
> >
> > wrote:
> >>
> >> Hello,
> >>
> >> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
> >> master on a specific node in a YARN cluster in order to open a server
> >> on a predetermined IP address and port. To that end, I wrote a driver
> >> program in which I created a ResourceRequest object and called
> >> setResourceName method to set a hostname, and attached it to a
> >> ApplicationSubmissionContext object by
> >> callingsetAMContainerResourceRequest method.
> >>
> >> I tried several times but couldn't launch the app master on a specific
> >> node. After searching code, I found that RMAppAttemptImpl invalidates
> >> what I've set in ResourceRequest as follows:
> >>
> >> // Currently, following fields are all hard code,
> >> // TODO: change these fields when we want to support
> >> // priority/resource-name/relax-locality specification for AM
> >> containers
> >> // allocation.
> >> appAttempt.amReq.setNumContainers(1);
> >> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
> >> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
> >> appAttempt.amReq.setRelaxLocality(true);
> >>
> >> Is there another way to launch a container for an application master
> >> on a specific node in Hadoop-2.6.0?
> >>
> >> Thanks.
> >>
> >> Dongwon Kim
> >
> >
>