You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Dongwon Kim <ea...@postech.ac.kr> on 2015/03/30 04:04:32 UTC
Run my own application master on a specific node in a YARN cluster
Hello,
First of all, I'm using Hadoop-2.6.0. I want to launch my own app
master on a specific node in a YARN cluster in order to open a server
on a predetermined IP address and port. To that end, I wrote a driver
program in which I created a ResourceRequest object and called
setResourceName method to set a hostname, and attached it to a
ApplicationSubmissionContext object by
callingsetAMContainerResourceRequest method.
I tried several times but couldn't launch the app master on a specific
node. After searching code, I found that RMAppAttemptImpl invalidates
what I've set in ResourceRequest as follows:
// Currently, following fields are all hard code,
// TODO: change these fields when we want to support
// priority/resource-name/relax-locality specification for AM
containers
// allocation.
appAttempt.amReq.setNumContainers(1);
appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
appAttempt.amReq.setResourceName(ResourceRequest.ANY);
appAttempt.amReq.setRelaxLocality(true);
Is there another way to launch a container for an application master
on a specific node in Hadoop-2.6.0?
Thanks.
Dongwon Kim
Re: Run my own application master on a specific node in a YARN cluster
Posted by Dongwon Kim <ea...@postech.ac.kr>.
I'm doing precisely the opposite.
My own node manager (MY_NM) is an AM in YARN and, therefore, each
MY_NM is expected to run inside a YARN container.
What I trying to do is to execute the AM (MY_NM) on each slave.
For the reason, I need to launch an AM on a specific node but
Hadoop-2.6.0 ignores what I describe in a ResourceRequest object.
According to what I've understood, RMAppManager in ResourceManager
creates a RMAppImpl object for each application and a RMAppAttempt
object for each application attempt.
When the state of RMAppAttempt changes from SUBMITTED to SCHEDULED,
RMAppAttempt invalidates necessary information to launch a container
for AM such as # containers, priority, resource name (=hostname),
whether to relax locality as in the above code and then asks
YarnScheduler to allocate a container.
I conclude that any application master cannot be launched on a
specific node in Hadoop-2.6.0.
As a workaround, I try to execute an AM directly on a specific node
from the command line without submitting it to YARN and register the
AM to RM.
It however fails (of course :P) because the AM is not given a
delegation token necessary to communicate with a RPC server in RM.
Anyway I need to find another way of doing that or I have to alter the
design of my framework.
Thanks.
- Dongwon Kim
2015-04-01 17:50 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> Very interesting, BTW. So you try to launch app-master with YARN Container
> but your own node-manager without YARN Container, Am I right?
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Wed, Apr 1, 2015 at 3:38 PM, Dongwon Kim <ea...@postech.ac.kr>
> wrote:
>>
>> Thanks for your input but I need to launch my own node manager
>> (different from the Yarn NM) running on each node.
>> (which is not explained in the original question)
>>
>> If I were to launch just a single master with a well-known address,
>> ZooKeeper would be a great solution!
>> Thanks.
>>
>> Dongwon Kim
>>
>> 2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
>> > Hi,
>> >
>> > In these circumstances, there is no easy way to do that. Maybe use
>> > workaround. How about using zookeeper for shared storage? The app master
>> > create predefined zookeeper node when starting with current machine's IP
>> > and
>> > Clients always look for that zookeeper node for app master's location.
>> >
>> > Thanks.
>> >
>> >
>> > Drake 민영근 Ph.D
>> > kt NexR
>> >
>> > On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim
>> > <ea...@postech.ac.kr>
>> > wrote:
>> >>
>> >> Hello,
>> >>
>> >> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
>> >> master on a specific node in a YARN cluster in order to open a server
>> >> on a predetermined IP address and port. To that end, I wrote a driver
>> >> program in which I created a ResourceRequest object and called
>> >> setResourceName method to set a hostname, and attached it to a
>> >> ApplicationSubmissionContext object by
>> >> callingsetAMContainerResourceRequest method.
>> >>
>> >> I tried several times but couldn't launch the app master on a specific
>> >> node. After searching code, I found that RMAppAttemptImpl invalidates
>> >> what I've set in ResourceRequest as follows:
>> >>
>> >> // Currently, following fields are all hard code,
>> >> // TODO: change these fields when we want to support
>> >> // priority/resource-name/relax-locality specification for AM
>> >> containers
>> >> // allocation.
>> >> appAttempt.amReq.setNumContainers(1);
>> >> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
>> >> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
>> >> appAttempt.amReq.setRelaxLocality(true);
>> >>
>> >> Is there another way to launch a container for an application master
>> >> on a specific node in Hadoop-2.6.0?
>> >>
>> >> Thanks.
>> >>
>> >> Dongwon Kim
>> >
>> >
>
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Dongwon Kim <ea...@postech.ac.kr>.
I'm doing precisely the opposite.
My own node manager (MY_NM) is an AM in YARN and, therefore, each
MY_NM is expected to run inside a YARN container.
What I trying to do is to execute the AM (MY_NM) on each slave.
For the reason, I need to launch an AM on a specific node but
Hadoop-2.6.0 ignores what I describe in a ResourceRequest object.
According to what I've understood, RMAppManager in ResourceManager
creates a RMAppImpl object for each application and a RMAppAttempt
object for each application attempt.
When the state of RMAppAttempt changes from SUBMITTED to SCHEDULED,
RMAppAttempt invalidates necessary information to launch a container
for AM such as # containers, priority, resource name (=hostname),
whether to relax locality as in the above code and then asks
YarnScheduler to allocate a container.
I conclude that any application master cannot be launched on a
specific node in Hadoop-2.6.0.
As a workaround, I try to execute an AM directly on a specific node
from the command line without submitting it to YARN and register the
AM to RM.
It however fails (of course :P) because the AM is not given a
delegation token necessary to communicate with a RPC server in RM.
Anyway I need to find another way of doing that or I have to alter the
design of my framework.
Thanks.
- Dongwon Kim
2015-04-01 17:50 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> Very interesting, BTW. So you try to launch app-master with YARN Container
> but your own node-manager without YARN Container, Am I right?
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Wed, Apr 1, 2015 at 3:38 PM, Dongwon Kim <ea...@postech.ac.kr>
> wrote:
>>
>> Thanks for your input but I need to launch my own node manager
>> (different from the Yarn NM) running on each node.
>> (which is not explained in the original question)
>>
>> If I were to launch just a single master with a well-known address,
>> ZooKeeper would be a great solution!
>> Thanks.
>>
>> Dongwon Kim
>>
>> 2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
>> > Hi,
>> >
>> > In these circumstances, there is no easy way to do that. Maybe use
>> > workaround. How about using zookeeper for shared storage? The app master
>> > create predefined zookeeper node when starting with current machine's IP
>> > and
>> > Clients always look for that zookeeper node for app master's location.
>> >
>> > Thanks.
>> >
>> >
>> > Drake 민영근 Ph.D
>> > kt NexR
>> >
>> > On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim
>> > <ea...@postech.ac.kr>
>> > wrote:
>> >>
>> >> Hello,
>> >>
>> >> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
>> >> master on a specific node in a YARN cluster in order to open a server
>> >> on a predetermined IP address and port. To that end, I wrote a driver
>> >> program in which I created a ResourceRequest object and called
>> >> setResourceName method to set a hostname, and attached it to a
>> >> ApplicationSubmissionContext object by
>> >> callingsetAMContainerResourceRequest method.
>> >>
>> >> I tried several times but couldn't launch the app master on a specific
>> >> node. After searching code, I found that RMAppAttemptImpl invalidates
>> >> what I've set in ResourceRequest as follows:
>> >>
>> >> // Currently, following fields are all hard code,
>> >> // TODO: change these fields when we want to support
>> >> // priority/resource-name/relax-locality specification for AM
>> >> containers
>> >> // allocation.
>> >> appAttempt.amReq.setNumContainers(1);
>> >> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
>> >> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
>> >> appAttempt.amReq.setRelaxLocality(true);
>> >>
>> >> Is there another way to launch a container for an application master
>> >> on a specific node in Hadoop-2.6.0?
>> >>
>> >> Thanks.
>> >>
>> >> Dongwon Kim
>> >
>> >
>
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Dongwon Kim <ea...@postech.ac.kr>.
I'm doing precisely the opposite.
My own node manager (MY_NM) is an AM in YARN and, therefore, each
MY_NM is expected to run inside a YARN container.
What I trying to do is to execute the AM (MY_NM) on each slave.
For the reason, I need to launch an AM on a specific node but
Hadoop-2.6.0 ignores what I describe in a ResourceRequest object.
According to what I've understood, RMAppManager in ResourceManager
creates a RMAppImpl object for each application and a RMAppAttempt
object for each application attempt.
When the state of RMAppAttempt changes from SUBMITTED to SCHEDULED,
RMAppAttempt invalidates necessary information to launch a container
for AM such as # containers, priority, resource name (=hostname),
whether to relax locality as in the above code and then asks
YarnScheduler to allocate a container.
I conclude that any application master cannot be launched on a
specific node in Hadoop-2.6.0.
As a workaround, I try to execute an AM directly on a specific node
from the command line without submitting it to YARN and register the
AM to RM.
It however fails (of course :P) because the AM is not given a
delegation token necessary to communicate with a RPC server in RM.
Anyway I need to find another way of doing that or I have to alter the
design of my framework.
Thanks.
- Dongwon Kim
2015-04-01 17:50 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> Very interesting, BTW. So you try to launch app-master with YARN Container
> but your own node-manager without YARN Container, Am I right?
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Wed, Apr 1, 2015 at 3:38 PM, Dongwon Kim <ea...@postech.ac.kr>
> wrote:
>>
>> Thanks for your input but I need to launch my own node manager
>> (different from the Yarn NM) running on each node.
>> (which is not explained in the original question)
>>
>> If I were to launch just a single master with a well-known address,
>> ZooKeeper would be a great solution!
>> Thanks.
>>
>> Dongwon Kim
>>
>> 2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
>> > Hi,
>> >
>> > In these circumstances, there is no easy way to do that. Maybe use
>> > workaround. How about using zookeeper for shared storage? The app master
>> > create predefined zookeeper node when starting with current machine's IP
>> > and
>> > Clients always look for that zookeeper node for app master's location.
>> >
>> > Thanks.
>> >
>> >
>> > Drake 민영근 Ph.D
>> > kt NexR
>> >
>> > On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim
>> > <ea...@postech.ac.kr>
>> > wrote:
>> >>
>> >> Hello,
>> >>
>> >> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
>> >> master on a specific node in a YARN cluster in order to open a server
>> >> on a predetermined IP address and port. To that end, I wrote a driver
>> >> program in which I created a ResourceRequest object and called
>> >> setResourceName method to set a hostname, and attached it to a
>> >> ApplicationSubmissionContext object by
>> >> callingsetAMContainerResourceRequest method.
>> >>
>> >> I tried several times but couldn't launch the app master on a specific
>> >> node. After searching code, I found that RMAppAttemptImpl invalidates
>> >> what I've set in ResourceRequest as follows:
>> >>
>> >> // Currently, following fields are all hard code,
>> >> // TODO: change these fields when we want to support
>> >> // priority/resource-name/relax-locality specification for AM
>> >> containers
>> >> // allocation.
>> >> appAttempt.amReq.setNumContainers(1);
>> >> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
>> >> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
>> >> appAttempt.amReq.setRelaxLocality(true);
>> >>
>> >> Is there another way to launch a container for an application master
>> >> on a specific node in Hadoop-2.6.0?
>> >>
>> >> Thanks.
>> >>
>> >> Dongwon Kim
>> >
>> >
>
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Dongwon Kim <ea...@postech.ac.kr>.
I'm doing precisely the opposite.
My own node manager (MY_NM) is an AM in YARN and, therefore, each
MY_NM is expected to run inside a YARN container.
What I trying to do is to execute the AM (MY_NM) on each slave.
For the reason, I need to launch an AM on a specific node but
Hadoop-2.6.0 ignores what I describe in a ResourceRequest object.
According to what I've understood, RMAppManager in ResourceManager
creates a RMAppImpl object for each application and a RMAppAttempt
object for each application attempt.
When the state of RMAppAttempt changes from SUBMITTED to SCHEDULED,
RMAppAttempt invalidates necessary information to launch a container
for AM such as # containers, priority, resource name (=hostname),
whether to relax locality as in the above code and then asks
YarnScheduler to allocate a container.
I conclude that any application master cannot be launched on a
specific node in Hadoop-2.6.0.
As a workaround, I try to execute an AM directly on a specific node
from the command line without submitting it to YARN and register the
AM to RM.
It however fails (of course :P) because the AM is not given a
delegation token necessary to communicate with a RPC server in RM.
Anyway I need to find another way of doing that or I have to alter the
design of my framework.
Thanks.
- Dongwon Kim
2015-04-01 17:50 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> Very interesting, BTW. So you try to launch app-master with YARN Container
> but your own node-manager without YARN Container, Am I right?
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Wed, Apr 1, 2015 at 3:38 PM, Dongwon Kim <ea...@postech.ac.kr>
> wrote:
>>
>> Thanks for your input but I need to launch my own node manager
>> (different from the Yarn NM) running on each node.
>> (which is not explained in the original question)
>>
>> If I were to launch just a single master with a well-known address,
>> ZooKeeper would be a great solution!
>> Thanks.
>>
>> Dongwon Kim
>>
>> 2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
>> > Hi,
>> >
>> > In these circumstances, there is no easy way to do that. Maybe use
>> > workaround. How about using zookeeper for shared storage? The app master
>> > create predefined zookeeper node when starting with current machine's IP
>> > and
>> > Clients always look for that zookeeper node for app master's location.
>> >
>> > Thanks.
>> >
>> >
>> > Drake 민영근 Ph.D
>> > kt NexR
>> >
>> > On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim
>> > <ea...@postech.ac.kr>
>> > wrote:
>> >>
>> >> Hello,
>> >>
>> >> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
>> >> master on a specific node in a YARN cluster in order to open a server
>> >> on a predetermined IP address and port. To that end, I wrote a driver
>> >> program in which I created a ResourceRequest object and called
>> >> setResourceName method to set a hostname, and attached it to a
>> >> ApplicationSubmissionContext object by
>> >> callingsetAMContainerResourceRequest method.
>> >>
>> >> I tried several times but couldn't launch the app master on a specific
>> >> node. After searching code, I found that RMAppAttemptImpl invalidates
>> >> what I've set in ResourceRequest as follows:
>> >>
>> >> // Currently, following fields are all hard code,
>> >> // TODO: change these fields when we want to support
>> >> // priority/resource-name/relax-locality specification for AM
>> >> containers
>> >> // allocation.
>> >> appAttempt.amReq.setNumContainers(1);
>> >> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
>> >> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
>> >> appAttempt.amReq.setRelaxLocality(true);
>> >>
>> >> Is there another way to launch a container for an application master
>> >> on a specific node in Hadoop-2.6.0?
>> >>
>> >> Thanks.
>> >>
>> >> Dongwon Kim
>> >
>> >
>
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Drake민영근 <dr...@nexr.com>.
Very interesting, BTW. So you try to launch app-master with YARN Container
but your own node-manager without YARN Container, Am I right?
Drake 민영근 Ph.D
kt NexR
On Wed, Apr 1, 2015 at 3:38 PM, Dongwon Kim <ea...@postech.ac.kr>
wrote:
> Thanks for your input but I need to launch my own node manager
> (different from the Yarn NM) running on each node.
> (which is not explained in the original question)
>
> If I were to launch just a single master with a well-known address,
> ZooKeeper would be a great solution!
> Thanks.
>
> Dongwon Kim
>
> 2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> > Hi,
> >
> > In these circumstances, there is no easy way to do that. Maybe use
> > workaround. How about using zookeeper for shared storage? The app master
> > create predefined zookeeper node when starting with current machine's IP
> and
> > Clients always look for that zookeeper node for app master's location.
> >
> > Thanks.
> >
> >
> > Drake 민영근 Ph.D
> > kt NexR
> >
> > On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim <eastcirclek@postech.ac.kr
> >
> > wrote:
> >>
> >> Hello,
> >>
> >> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
> >> master on a specific node in a YARN cluster in order to open a server
> >> on a predetermined IP address and port. To that end, I wrote a driver
> >> program in which I created a ResourceRequest object and called
> >> setResourceName method to set a hostname, and attached it to a
> >> ApplicationSubmissionContext object by
> >> callingsetAMContainerResourceRequest method.
> >>
> >> I tried several times but couldn't launch the app master on a specific
> >> node. After searching code, I found that RMAppAttemptImpl invalidates
> >> what I've set in ResourceRequest as follows:
> >>
> >> // Currently, following fields are all hard code,
> >> // TODO: change these fields when we want to support
> >> // priority/resource-name/relax-locality specification for AM
> >> containers
> >> // allocation.
> >> appAttempt.amReq.setNumContainers(1);
> >> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
> >> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
> >> appAttempt.amReq.setRelaxLocality(true);
> >>
> >> Is there another way to launch a container for an application master
> >> on a specific node in Hadoop-2.6.0?
> >>
> >> Thanks.
> >>
> >> Dongwon Kim
> >
> >
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Drake민영근 <dr...@nexr.com>.
Very interesting, BTW. So you try to launch app-master with YARN Container
but your own node-manager without YARN Container, Am I right?
Drake 민영근 Ph.D
kt NexR
On Wed, Apr 1, 2015 at 3:38 PM, Dongwon Kim <ea...@postech.ac.kr>
wrote:
> Thanks for your input but I need to launch my own node manager
> (different from the Yarn NM) running on each node.
> (which is not explained in the original question)
>
> If I were to launch just a single master with a well-known address,
> ZooKeeper would be a great solution!
> Thanks.
>
> Dongwon Kim
>
> 2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> > Hi,
> >
> > In these circumstances, there is no easy way to do that. Maybe use
> > workaround. How about using zookeeper for shared storage? The app master
> > create predefined zookeeper node when starting with current machine's IP
> and
> > Clients always look for that zookeeper node for app master's location.
> >
> > Thanks.
> >
> >
> > Drake 민영근 Ph.D
> > kt NexR
> >
> > On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim <eastcirclek@postech.ac.kr
> >
> > wrote:
> >>
> >> Hello,
> >>
> >> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
> >> master on a specific node in a YARN cluster in order to open a server
> >> on a predetermined IP address and port. To that end, I wrote a driver
> >> program in which I created a ResourceRequest object and called
> >> setResourceName method to set a hostname, and attached it to a
> >> ApplicationSubmissionContext object by
> >> callingsetAMContainerResourceRequest method.
> >>
> >> I tried several times but couldn't launch the app master on a specific
> >> node. After searching code, I found that RMAppAttemptImpl invalidates
> >> what I've set in ResourceRequest as follows:
> >>
> >> // Currently, following fields are all hard code,
> >> // TODO: change these fields when we want to support
> >> // priority/resource-name/relax-locality specification for AM
> >> containers
> >> // allocation.
> >> appAttempt.amReq.setNumContainers(1);
> >> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
> >> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
> >> appAttempt.amReq.setRelaxLocality(true);
> >>
> >> Is there another way to launch a container for an application master
> >> on a specific node in Hadoop-2.6.0?
> >>
> >> Thanks.
> >>
> >> Dongwon Kim
> >
> >
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Drake민영근 <dr...@nexr.com>.
Very interesting, BTW. So you try to launch app-master with YARN Container
but your own node-manager without YARN Container, Am I right?
Drake 민영근 Ph.D
kt NexR
On Wed, Apr 1, 2015 at 3:38 PM, Dongwon Kim <ea...@postech.ac.kr>
wrote:
> Thanks for your input but I need to launch my own node manager
> (different from the Yarn NM) running on each node.
> (which is not explained in the original question)
>
> If I were to launch just a single master with a well-known address,
> ZooKeeper would be a great solution!
> Thanks.
>
> Dongwon Kim
>
> 2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> > Hi,
> >
> > In these circumstances, there is no easy way to do that. Maybe use
> > workaround. How about using zookeeper for shared storage? The app master
> > create predefined zookeeper node when starting with current machine's IP
> and
> > Clients always look for that zookeeper node for app master's location.
> >
> > Thanks.
> >
> >
> > Drake 민영근 Ph.D
> > kt NexR
> >
> > On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim <eastcirclek@postech.ac.kr
> >
> > wrote:
> >>
> >> Hello,
> >>
> >> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
> >> master on a specific node in a YARN cluster in order to open a server
> >> on a predetermined IP address and port. To that end, I wrote a driver
> >> program in which I created a ResourceRequest object and called
> >> setResourceName method to set a hostname, and attached it to a
> >> ApplicationSubmissionContext object by
> >> callingsetAMContainerResourceRequest method.
> >>
> >> I tried several times but couldn't launch the app master on a specific
> >> node. After searching code, I found that RMAppAttemptImpl invalidates
> >> what I've set in ResourceRequest as follows:
> >>
> >> // Currently, following fields are all hard code,
> >> // TODO: change these fields when we want to support
> >> // priority/resource-name/relax-locality specification for AM
> >> containers
> >> // allocation.
> >> appAttempt.amReq.setNumContainers(1);
> >> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
> >> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
> >> appAttempt.amReq.setRelaxLocality(true);
> >>
> >> Is there another way to launch a container for an application master
> >> on a specific node in Hadoop-2.6.0?
> >>
> >> Thanks.
> >>
> >> Dongwon Kim
> >
> >
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Drake민영근 <dr...@nexr.com>.
Very interesting, BTW. So you try to launch app-master with YARN Container
but your own node-manager without YARN Container, Am I right?
Drake 민영근 Ph.D
kt NexR
On Wed, Apr 1, 2015 at 3:38 PM, Dongwon Kim <ea...@postech.ac.kr>
wrote:
> Thanks for your input but I need to launch my own node manager
> (different from the Yarn NM) running on each node.
> (which is not explained in the original question)
>
> If I were to launch just a single master with a well-known address,
> ZooKeeper would be a great solution!
> Thanks.
>
> Dongwon Kim
>
> 2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> > Hi,
> >
> > In these circumstances, there is no easy way to do that. Maybe use
> > workaround. How about using zookeeper for shared storage? The app master
> > create predefined zookeeper node when starting with current machine's IP
> and
> > Clients always look for that zookeeper node for app master's location.
> >
> > Thanks.
> >
> >
> > Drake 민영근 Ph.D
> > kt NexR
> >
> > On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim <eastcirclek@postech.ac.kr
> >
> > wrote:
> >>
> >> Hello,
> >>
> >> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
> >> master on a specific node in a YARN cluster in order to open a server
> >> on a predetermined IP address and port. To that end, I wrote a driver
> >> program in which I created a ResourceRequest object and called
> >> setResourceName method to set a hostname, and attached it to a
> >> ApplicationSubmissionContext object by
> >> callingsetAMContainerResourceRequest method.
> >>
> >> I tried several times but couldn't launch the app master on a specific
> >> node. After searching code, I found that RMAppAttemptImpl invalidates
> >> what I've set in ResourceRequest as follows:
> >>
> >> // Currently, following fields are all hard code,
> >> // TODO: change these fields when we want to support
> >> // priority/resource-name/relax-locality specification for AM
> >> containers
> >> // allocation.
> >> appAttempt.amReq.setNumContainers(1);
> >> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
> >> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
> >> appAttempt.amReq.setRelaxLocality(true);
> >>
> >> Is there another way to launch a container for an application master
> >> on a specific node in Hadoop-2.6.0?
> >>
> >> Thanks.
> >>
> >> Dongwon Kim
> >
> >
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Dongwon Kim <ea...@postech.ac.kr>.
Thanks for your input but I need to launch my own node manager
(different from the Yarn NM) running on each node.
(which is not explained in the original question)
If I were to launch just a single master with a well-known address,
ZooKeeper would be a great solution!
Thanks.
Dongwon Kim
2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> Hi,
>
> In these circumstances, there is no easy way to do that. Maybe use
> workaround. How about using zookeeper for shared storage? The app master
> create predefined zookeeper node when starting with current machine's IP and
> Clients always look for that zookeeper node for app master's location.
>
> Thanks.
>
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim <ea...@postech.ac.kr>
> wrote:
>>
>> Hello,
>>
>> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
>> master on a specific node in a YARN cluster in order to open a server
>> on a predetermined IP address and port. To that end, I wrote a driver
>> program in which I created a ResourceRequest object and called
>> setResourceName method to set a hostname, and attached it to a
>> ApplicationSubmissionContext object by
>> callingsetAMContainerResourceRequest method.
>>
>> I tried several times but couldn't launch the app master on a specific
>> node. After searching code, I found that RMAppAttemptImpl invalidates
>> what I've set in ResourceRequest as follows:
>>
>> // Currently, following fields are all hard code,
>> // TODO: change these fields when we want to support
>> // priority/resource-name/relax-locality specification for AM
>> containers
>> // allocation.
>> appAttempt.amReq.setNumContainers(1);
>> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
>> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
>> appAttempt.amReq.setRelaxLocality(true);
>>
>> Is there another way to launch a container for an application master
>> on a specific node in Hadoop-2.6.0?
>>
>> Thanks.
>>
>> Dongwon Kim
>
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Dongwon Kim <ea...@postech.ac.kr>.
Thanks for your input but I need to launch my own node manager
(different from the Yarn NM) running on each node.
(which is not explained in the original question)
If I were to launch just a single master with a well-known address,
ZooKeeper would be a great solution!
Thanks.
Dongwon Kim
2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> Hi,
>
> In these circumstances, there is no easy way to do that. Maybe use
> workaround. How about using zookeeper for shared storage? The app master
> create predefined zookeeper node when starting with current machine's IP and
> Clients always look for that zookeeper node for app master's location.
>
> Thanks.
>
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim <ea...@postech.ac.kr>
> wrote:
>>
>> Hello,
>>
>> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
>> master on a specific node in a YARN cluster in order to open a server
>> on a predetermined IP address and port. To that end, I wrote a driver
>> program in which I created a ResourceRequest object and called
>> setResourceName method to set a hostname, and attached it to a
>> ApplicationSubmissionContext object by
>> callingsetAMContainerResourceRequest method.
>>
>> I tried several times but couldn't launch the app master on a specific
>> node. After searching code, I found that RMAppAttemptImpl invalidates
>> what I've set in ResourceRequest as follows:
>>
>> // Currently, following fields are all hard code,
>> // TODO: change these fields when we want to support
>> // priority/resource-name/relax-locality specification for AM
>> containers
>> // allocation.
>> appAttempt.amReq.setNumContainers(1);
>> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
>> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
>> appAttempt.amReq.setRelaxLocality(true);
>>
>> Is there another way to launch a container for an application master
>> on a specific node in Hadoop-2.6.0?
>>
>> Thanks.
>>
>> Dongwon Kim
>
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Dongwon Kim <ea...@postech.ac.kr>.
Thanks for your input but I need to launch my own node manager
(different from the Yarn NM) running on each node.
(which is not explained in the original question)
If I were to launch just a single master with a well-known address,
ZooKeeper would be a great solution!
Thanks.
Dongwon Kim
2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> Hi,
>
> In these circumstances, there is no easy way to do that. Maybe use
> workaround. How about using zookeeper for shared storage? The app master
> create predefined zookeeper node when starting with current machine's IP and
> Clients always look for that zookeeper node for app master's location.
>
> Thanks.
>
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim <ea...@postech.ac.kr>
> wrote:
>>
>> Hello,
>>
>> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
>> master on a specific node in a YARN cluster in order to open a server
>> on a predetermined IP address and port. To that end, I wrote a driver
>> program in which I created a ResourceRequest object and called
>> setResourceName method to set a hostname, and attached it to a
>> ApplicationSubmissionContext object by
>> callingsetAMContainerResourceRequest method.
>>
>> I tried several times but couldn't launch the app master on a specific
>> node. After searching code, I found that RMAppAttemptImpl invalidates
>> what I've set in ResourceRequest as follows:
>>
>> // Currently, following fields are all hard code,
>> // TODO: change these fields when we want to support
>> // priority/resource-name/relax-locality specification for AM
>> containers
>> // allocation.
>> appAttempt.amReq.setNumContainers(1);
>> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
>> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
>> appAttempt.amReq.setRelaxLocality(true);
>>
>> Is there another way to launch a container for an application master
>> on a specific node in Hadoop-2.6.0?
>>
>> Thanks.
>>
>> Dongwon Kim
>
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Dongwon Kim <ea...@postech.ac.kr>.
Thanks for your input but I need to launch my own node manager
(different from the Yarn NM) running on each node.
(which is not explained in the original question)
If I were to launch just a single master with a well-known address,
ZooKeeper would be a great solution!
Thanks.
Dongwon Kim
2015-03-31 10:47 GMT+09:00 Drake민영근 <dr...@nexr.com>:
> Hi,
>
> In these circumstances, there is no easy way to do that. Maybe use
> workaround. How about using zookeeper for shared storage? The app master
> create predefined zookeeper node when starting with current machine's IP and
> Clients always look for that zookeeper node for app master's location.
>
> Thanks.
>
>
> Drake 민영근 Ph.D
> kt NexR
>
> On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim <ea...@postech.ac.kr>
> wrote:
>>
>> Hello,
>>
>> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
>> master on a specific node in a YARN cluster in order to open a server
>> on a predetermined IP address and port. To that end, I wrote a driver
>> program in which I created a ResourceRequest object and called
>> setResourceName method to set a hostname, and attached it to a
>> ApplicationSubmissionContext object by
>> callingsetAMContainerResourceRequest method.
>>
>> I tried several times but couldn't launch the app master on a specific
>> node. After searching code, I found that RMAppAttemptImpl invalidates
>> what I've set in ResourceRequest as follows:
>>
>> // Currently, following fields are all hard code,
>> // TODO: change these fields when we want to support
>> // priority/resource-name/relax-locality specification for AM
>> containers
>> // allocation.
>> appAttempt.amReq.setNumContainers(1);
>> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
>> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
>> appAttempt.amReq.setRelaxLocality(true);
>>
>> Is there another way to launch a container for an application master
>> on a specific node in Hadoop-2.6.0?
>>
>> Thanks.
>>
>> Dongwon Kim
>
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Drake민영근 <dr...@nexr.com>.
Hi,
In these circumstances, there is no easy way to do that. Maybe use
workaround. How about using zookeeper for shared storage? The app master
create predefined zookeeper node when starting with current machine's IP
and Clients always look for that zookeeper node for app master's location.
Thanks.
Drake 민영근 Ph.D
kt NexR
On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim <ea...@postech.ac.kr>
wrote:
> Hello,
>
> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
> master on a specific node in a YARN cluster in order to open a server
> on a predetermined IP address and port. To that end, I wrote a driver
> program in which I created a ResourceRequest object and called
> setResourceName method to set a hostname, and attached it to a
> ApplicationSubmissionContext object by
> callingsetAMContainerResourceRequest method.
>
> I tried several times but couldn't launch the app master on a specific
> node. After searching code, I found that RMAppAttemptImpl invalidates
> what I've set in ResourceRequest as follows:
>
> // Currently, following fields are all hard code,
> // TODO: change these fields when we want to support
> // priority/resource-name/relax-locality specification for AM
> containers
> // allocation.
> appAttempt.amReq.setNumContainers(1);
> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
> appAttempt.amReq.setRelaxLocality(true);
>
> Is there another way to launch a container for an application master
> on a specific node in Hadoop-2.6.0?
>
> Thanks.
>
> Dongwon Kim
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Drake민영근 <dr...@nexr.com>.
Hi,
In these circumstances, there is no easy way to do that. Maybe use
workaround. How about using zookeeper for shared storage? The app master
create predefined zookeeper node when starting with current machine's IP
and Clients always look for that zookeeper node for app master's location.
Thanks.
Drake 민영근 Ph.D
kt NexR
On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim <ea...@postech.ac.kr>
wrote:
> Hello,
>
> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
> master on a specific node in a YARN cluster in order to open a server
> on a predetermined IP address and port. To that end, I wrote a driver
> program in which I created a ResourceRequest object and called
> setResourceName method to set a hostname, and attached it to a
> ApplicationSubmissionContext object by
> callingsetAMContainerResourceRequest method.
>
> I tried several times but couldn't launch the app master on a specific
> node. After searching code, I found that RMAppAttemptImpl invalidates
> what I've set in ResourceRequest as follows:
>
> // Currently, following fields are all hard code,
> // TODO: change these fields when we want to support
> // priority/resource-name/relax-locality specification for AM
> containers
> // allocation.
> appAttempt.amReq.setNumContainers(1);
> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
> appAttempt.amReq.setRelaxLocality(true);
>
> Is there another way to launch a container for an application master
> on a specific node in Hadoop-2.6.0?
>
> Thanks.
>
> Dongwon Kim
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Drake민영근 <dr...@nexr.com>.
Hi,
In these circumstances, there is no easy way to do that. Maybe use
workaround. How about using zookeeper for shared storage? The app master
create predefined zookeeper node when starting with current machine's IP
and Clients always look for that zookeeper node for app master's location.
Thanks.
Drake 민영근 Ph.D
kt NexR
On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim <ea...@postech.ac.kr>
wrote:
> Hello,
>
> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
> master on a specific node in a YARN cluster in order to open a server
> on a predetermined IP address and port. To that end, I wrote a driver
> program in which I created a ResourceRequest object and called
> setResourceName method to set a hostname, and attached it to a
> ApplicationSubmissionContext object by
> callingsetAMContainerResourceRequest method.
>
> I tried several times but couldn't launch the app master on a specific
> node. After searching code, I found that RMAppAttemptImpl invalidates
> what I've set in ResourceRequest as follows:
>
> // Currently, following fields are all hard code,
> // TODO: change these fields when we want to support
> // priority/resource-name/relax-locality specification for AM
> containers
> // allocation.
> appAttempt.amReq.setNumContainers(1);
> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
> appAttempt.amReq.setRelaxLocality(true);
>
> Is there another way to launch a container for an application master
> on a specific node in Hadoop-2.6.0?
>
> Thanks.
>
> Dongwon Kim
>
Re: Run my own application master on a specific node in a YARN cluster
Posted by Drake민영근 <dr...@nexr.com>.
Hi,
In these circumstances, there is no easy way to do that. Maybe use
workaround. How about using zookeeper for shared storage? The app master
create predefined zookeeper node when starting with current machine's IP
and Clients always look for that zookeeper node for app master's location.
Thanks.
Drake 민영근 Ph.D
kt NexR
On Mon, Mar 30, 2015 at 11:04 AM, Dongwon Kim <ea...@postech.ac.kr>
wrote:
> Hello,
>
> First of all, I'm using Hadoop-2.6.0. I want to launch my own app
> master on a specific node in a YARN cluster in order to open a server
> on a predetermined IP address and port. To that end, I wrote a driver
> program in which I created a ResourceRequest object and called
> setResourceName method to set a hostname, and attached it to a
> ApplicationSubmissionContext object by
> callingsetAMContainerResourceRequest method.
>
> I tried several times but couldn't launch the app master on a specific
> node. After searching code, I found that RMAppAttemptImpl invalidates
> what I've set in ResourceRequest as follows:
>
> // Currently, following fields are all hard code,
> // TODO: change these fields when we want to support
> // priority/resource-name/relax-locality specification for AM
> containers
> // allocation.
> appAttempt.amReq.setNumContainers(1);
> appAttempt.amReq.setPriority(AM_CONTAINER_PRIORITY);
> appAttempt.amReq.setResourceName(ResourceRequest.ANY);
> appAttempt.amReq.setRelaxLocality(true);
>
> Is there another way to launch a container for an application master
> on a specific node in Hadoop-2.6.0?
>
> Thanks.
>
> Dongwon Kim
>