You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Anfernee Xu <an...@gmail.com> on 2014/02/13 20:34:08 UTC
job submission between 2 YARN clusters
I'm at Yarn 2.2.0 release, I configured 2 single-node clusters on my
laptop(just for POC and all port conflicts are resolved, and I can see NM
and RM is up, webUI shows everything is fine) and I also have a standalone
java application. The java application is a kind of job client, it will
submit job1 to Cluser #1, once the job is finished, it will submit another
job2 to Cluster #2.
What I'm seeing is the job1 is doing fine, but job2 failed, I looked source
code, and found the NM in cluser2 was talking to cluser1's RM via wrong
yarn.resourcemanager.scheduler.address. How that happens? I just want to
make sure there's no such issue in real deployment.
--
--Anfernee
Re: job submission between 2 YARN clusters
Posted by Zhijie Shen <zs...@hortonworks.com>.
I thought you need set the following configs differently from two cluster:
"yarn.resourcemanager.resource-tracker.address": NM talks to this address
"yarn.resourcemanager.scheduler.address": Your application talks to this
address
"yarn.resourcemanager.address": Your client talks to this address
Of course, NM needs to be started at different "yarn.nodemanager.address"
in your two clusters.
- Zhijie
On Thu, Feb 13, 2014 at 4:59 PM, Anfernee Xu <an...@gmail.com> wrote:
> Hi Zhijie,
>
> I agree, what I'm doing in the standalone app is that the app loads the
> first cluster Configuration(mapred-site.xml, yarn-site.xml) as its default
> configuration, and then submit MR job with this configuration to the first
> cluster, and after the job is finished, I will submit the second job to the
> second cluster with almost same Configuration exception I changed the
> property: yarn.resourcemanager.address pointing to the second cluster's RM.
> My guess the job.xml of the second job holds all property values of the
> first cluster(such as yarn.resourcemanager.scheduler.address) and will
> override these properties specified in the second cluster(yarn-site.xml for
> example), therefore it will talk to the wrong RM when NM is launching the
> container.
>
> Please comment.
>
> BTW, I just tweak the standalone app so that it will load the second
> cluster's configuration(yarn-site.xml) before submit the second job, it
> seems working.
>
> Thanks
>
>
> On Thu, Feb 13, 2014 at 4:28 PM, Zhijie Shen <zs...@hortonworks.com>wrote:
>
>> Hi Anfernee,
>>
>> It sounds most likely that config somehow corrupts. So you have two sets
>> of config to start two YARN cluster separately, don't you? If you provide
>> more detail about how you config the two clusters, it's easy for the
>> community to understand your problem.
>>
>> - Zhijie
>>
>>
>> On Thu, Feb 13, 2014 at 11:34 AM, Anfernee Xu <an...@gmail.com>wrote:
>>
>>> I'm at Yarn 2.2.0 release, I configured 2 single-node clusters on my
>>> laptop(just for POC and all port conflicts are resolved, and I can see NM
>>> and RM is up, webUI shows everything is fine) and I also have a standalone
>>> java application. The java application is a kind of job client, it will
>>> submit job1 to Cluser #1, once the job is finished, it will submit another
>>> job2 to Cluster #2.
>>>
>>> What I'm seeing is the job1 is doing fine, but job2 failed, I looked
>>> source code, and found the NM in cluser2 was talking to cluser1's RM via
>>> wrong yarn.resourcemanager.scheduler.address. How that happens? I just want
>>> to make sure there's no such issue in real deployment.
>>>
>>> --
>>> --Anfernee
>>>
>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
>
> --
> --Anfernee
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: job submission between 2 YARN clusters
Posted by Zhijie Shen <zs...@hortonworks.com>.
I thought you need set the following configs differently from two cluster:
"yarn.resourcemanager.resource-tracker.address": NM talks to this address
"yarn.resourcemanager.scheduler.address": Your application talks to this
address
"yarn.resourcemanager.address": Your client talks to this address
Of course, NM needs to be started at different "yarn.nodemanager.address"
in your two clusters.
- Zhijie
On Thu, Feb 13, 2014 at 4:59 PM, Anfernee Xu <an...@gmail.com> wrote:
> Hi Zhijie,
>
> I agree, what I'm doing in the standalone app is that the app loads the
> first cluster Configuration(mapred-site.xml, yarn-site.xml) as its default
> configuration, and then submit MR job with this configuration to the first
> cluster, and after the job is finished, I will submit the second job to the
> second cluster with almost same Configuration exception I changed the
> property: yarn.resourcemanager.address pointing to the second cluster's RM.
> My guess the job.xml of the second job holds all property values of the
> first cluster(such as yarn.resourcemanager.scheduler.address) and will
> override these properties specified in the second cluster(yarn-site.xml for
> example), therefore it will talk to the wrong RM when NM is launching the
> container.
>
> Please comment.
>
> BTW, I just tweak the standalone app so that it will load the second
> cluster's configuration(yarn-site.xml) before submit the second job, it
> seems working.
>
> Thanks
>
>
> On Thu, Feb 13, 2014 at 4:28 PM, Zhijie Shen <zs...@hortonworks.com>wrote:
>
>> Hi Anfernee,
>>
>> It sounds most likely that config somehow corrupts. So you have two sets
>> of config to start two YARN cluster separately, don't you? If you provide
>> more detail about how you config the two clusters, it's easy for the
>> community to understand your problem.
>>
>> - Zhijie
>>
>>
>> On Thu, Feb 13, 2014 at 11:34 AM, Anfernee Xu <an...@gmail.com>wrote:
>>
>>> I'm at Yarn 2.2.0 release, I configured 2 single-node clusters on my
>>> laptop(just for POC and all port conflicts are resolved, and I can see NM
>>> and RM is up, webUI shows everything is fine) and I also have a standalone
>>> java application. The java application is a kind of job client, it will
>>> submit job1 to Cluser #1, once the job is finished, it will submit another
>>> job2 to Cluster #2.
>>>
>>> What I'm seeing is the job1 is doing fine, but job2 failed, I looked
>>> source code, and found the NM in cluser2 was talking to cluser1's RM via
>>> wrong yarn.resourcemanager.scheduler.address. How that happens? I just want
>>> to make sure there's no such issue in real deployment.
>>>
>>> --
>>> --Anfernee
>>>
>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
>
> --
> --Anfernee
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: job submission between 2 YARN clusters
Posted by Zhijie Shen <zs...@hortonworks.com>.
I thought you need set the following configs differently from two cluster:
"yarn.resourcemanager.resource-tracker.address": NM talks to this address
"yarn.resourcemanager.scheduler.address": Your application talks to this
address
"yarn.resourcemanager.address": Your client talks to this address
Of course, NM needs to be started at different "yarn.nodemanager.address"
in your two clusters.
- Zhijie
On Thu, Feb 13, 2014 at 4:59 PM, Anfernee Xu <an...@gmail.com> wrote:
> Hi Zhijie,
>
> I agree, what I'm doing in the standalone app is that the app loads the
> first cluster Configuration(mapred-site.xml, yarn-site.xml) as its default
> configuration, and then submit MR job with this configuration to the first
> cluster, and after the job is finished, I will submit the second job to the
> second cluster with almost same Configuration exception I changed the
> property: yarn.resourcemanager.address pointing to the second cluster's RM.
> My guess the job.xml of the second job holds all property values of the
> first cluster(such as yarn.resourcemanager.scheduler.address) and will
> override these properties specified in the second cluster(yarn-site.xml for
> example), therefore it will talk to the wrong RM when NM is launching the
> container.
>
> Please comment.
>
> BTW, I just tweak the standalone app so that it will load the second
> cluster's configuration(yarn-site.xml) before submit the second job, it
> seems working.
>
> Thanks
>
>
> On Thu, Feb 13, 2014 at 4:28 PM, Zhijie Shen <zs...@hortonworks.com>wrote:
>
>> Hi Anfernee,
>>
>> It sounds most likely that config somehow corrupts. So you have two sets
>> of config to start two YARN cluster separately, don't you? If you provide
>> more detail about how you config the two clusters, it's easy for the
>> community to understand your problem.
>>
>> - Zhijie
>>
>>
>> On Thu, Feb 13, 2014 at 11:34 AM, Anfernee Xu <an...@gmail.com>wrote:
>>
>>> I'm at Yarn 2.2.0 release, I configured 2 single-node clusters on my
>>> laptop(just for POC and all port conflicts are resolved, and I can see NM
>>> and RM is up, webUI shows everything is fine) and I also have a standalone
>>> java application. The java application is a kind of job client, it will
>>> submit job1 to Cluser #1, once the job is finished, it will submit another
>>> job2 to Cluster #2.
>>>
>>> What I'm seeing is the job1 is doing fine, but job2 failed, I looked
>>> source code, and found the NM in cluser2 was talking to cluser1's RM via
>>> wrong yarn.resourcemanager.scheduler.address. How that happens? I just want
>>> to make sure there's no such issue in real deployment.
>>>
>>> --
>>> --Anfernee
>>>
>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
>
> --
> --Anfernee
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: job submission between 2 YARN clusters
Posted by Zhijie Shen <zs...@hortonworks.com>.
I thought you need set the following configs differently from two cluster:
"yarn.resourcemanager.resource-tracker.address": NM talks to this address
"yarn.resourcemanager.scheduler.address": Your application talks to this
address
"yarn.resourcemanager.address": Your client talks to this address
Of course, NM needs to be started at different "yarn.nodemanager.address"
in your two clusters.
- Zhijie
On Thu, Feb 13, 2014 at 4:59 PM, Anfernee Xu <an...@gmail.com> wrote:
> Hi Zhijie,
>
> I agree, what I'm doing in the standalone app is that the app loads the
> first cluster Configuration(mapred-site.xml, yarn-site.xml) as its default
> configuration, and then submit MR job with this configuration to the first
> cluster, and after the job is finished, I will submit the second job to the
> second cluster with almost same Configuration exception I changed the
> property: yarn.resourcemanager.address pointing to the second cluster's RM.
> My guess the job.xml of the second job holds all property values of the
> first cluster(such as yarn.resourcemanager.scheduler.address) and will
> override these properties specified in the second cluster(yarn-site.xml for
> example), therefore it will talk to the wrong RM when NM is launching the
> container.
>
> Please comment.
>
> BTW, I just tweak the standalone app so that it will load the second
> cluster's configuration(yarn-site.xml) before submit the second job, it
> seems working.
>
> Thanks
>
>
> On Thu, Feb 13, 2014 at 4:28 PM, Zhijie Shen <zs...@hortonworks.com>wrote:
>
>> Hi Anfernee,
>>
>> It sounds most likely that config somehow corrupts. So you have two sets
>> of config to start two YARN cluster separately, don't you? If you provide
>> more detail about how you config the two clusters, it's easy for the
>> community to understand your problem.
>>
>> - Zhijie
>>
>>
>> On Thu, Feb 13, 2014 at 11:34 AM, Anfernee Xu <an...@gmail.com>wrote:
>>
>>> I'm at Yarn 2.2.0 release, I configured 2 single-node clusters on my
>>> laptop(just for POC and all port conflicts are resolved, and I can see NM
>>> and RM is up, webUI shows everything is fine) and I also have a standalone
>>> java application. The java application is a kind of job client, it will
>>> submit job1 to Cluser #1, once the job is finished, it will submit another
>>> job2 to Cluster #2.
>>>
>>> What I'm seeing is the job1 is doing fine, but job2 failed, I looked
>>> source code, and found the NM in cluser2 was talking to cluser1's RM via
>>> wrong yarn.resourcemanager.scheduler.address. How that happens? I just want
>>> to make sure there's no such issue in real deployment.
>>>
>>> --
>>> --Anfernee
>>>
>>
>>
>>
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
>
> --
> --Anfernee
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: job submission between 2 YARN clusters
Posted by Anfernee Xu <an...@gmail.com>.
Hi Zhijie,
I agree, what I'm doing in the standalone app is that the app loads the
first cluster Configuration(mapred-site.xml, yarn-site.xml) as its default
configuration, and then submit MR job with this configuration to the first
cluster, and after the job is finished, I will submit the second job to the
second cluster with almost same Configuration exception I changed the
property: yarn.resourcemanager.address pointing to the second cluster's RM.
My guess the job.xml of the second job holds all property values of the
first cluster(such as yarn.resourcemanager.scheduler.address) and will
override these properties specified in the second cluster(yarn-site.xml for
example), therefore it will talk to the wrong RM when NM is launching the
container.
Please comment.
BTW, I just tweak the standalone app so that it will load the second
cluster's configuration(yarn-site.xml) before submit the second job, it
seems working.
Thanks
On Thu, Feb 13, 2014 at 4:28 PM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Hi Anfernee,
>
> It sounds most likely that config somehow corrupts. So you have two sets
> of config to start two YARN cluster separately, don't you? If you provide
> more detail about how you config the two clusters, it's easy for the
> community to understand your problem.
>
> - Zhijie
>
>
> On Thu, Feb 13, 2014 at 11:34 AM, Anfernee Xu <an...@gmail.com>wrote:
>
>> I'm at Yarn 2.2.0 release, I configured 2 single-node clusters on my
>> laptop(just for POC and all port conflicts are resolved, and I can see NM
>> and RM is up, webUI shows everything is fine) and I also have a standalone
>> java application. The java application is a kind of job client, it will
>> submit job1 to Cluser #1, once the job is finished, it will submit another
>> job2 to Cluster #2.
>>
>> What I'm seeing is the job1 is doing fine, but job2 failed, I looked
>> source code, and found the NM in cluser2 was talking to cluser1's RM via
>> wrong yarn.resourcemanager.scheduler.address. How that happens? I just want
>> to make sure there's no such issue in real deployment.
>>
>> --
>> --Anfernee
>>
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
--Anfernee
Re: job submission between 2 YARN clusters
Posted by Anfernee Xu <an...@gmail.com>.
Hi Zhijie,
I agree, what I'm doing in the standalone app is that the app loads the
first cluster Configuration(mapred-site.xml, yarn-site.xml) as its default
configuration, and then submit MR job with this configuration to the first
cluster, and after the job is finished, I will submit the second job to the
second cluster with almost same Configuration exception I changed the
property: yarn.resourcemanager.address pointing to the second cluster's RM.
My guess the job.xml of the second job holds all property values of the
first cluster(such as yarn.resourcemanager.scheduler.address) and will
override these properties specified in the second cluster(yarn-site.xml for
example), therefore it will talk to the wrong RM when NM is launching the
container.
Please comment.
BTW, I just tweak the standalone app so that it will load the second
cluster's configuration(yarn-site.xml) before submit the second job, it
seems working.
Thanks
On Thu, Feb 13, 2014 at 4:28 PM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Hi Anfernee,
>
> It sounds most likely that config somehow corrupts. So you have two sets
> of config to start two YARN cluster separately, don't you? If you provide
> more detail about how you config the two clusters, it's easy for the
> community to understand your problem.
>
> - Zhijie
>
>
> On Thu, Feb 13, 2014 at 11:34 AM, Anfernee Xu <an...@gmail.com>wrote:
>
>> I'm at Yarn 2.2.0 release, I configured 2 single-node clusters on my
>> laptop(just for POC and all port conflicts are resolved, and I can see NM
>> and RM is up, webUI shows everything is fine) and I also have a standalone
>> java application. The java application is a kind of job client, it will
>> submit job1 to Cluser #1, once the job is finished, it will submit another
>> job2 to Cluster #2.
>>
>> What I'm seeing is the job1 is doing fine, but job2 failed, I looked
>> source code, and found the NM in cluser2 was talking to cluser1's RM via
>> wrong yarn.resourcemanager.scheduler.address. How that happens? I just want
>> to make sure there's no such issue in real deployment.
>>
>> --
>> --Anfernee
>>
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
--Anfernee
Re: job submission between 2 YARN clusters
Posted by Anfernee Xu <an...@gmail.com>.
Hi Zhijie,
I agree, what I'm doing in the standalone app is that the app loads the
first cluster Configuration(mapred-site.xml, yarn-site.xml) as its default
configuration, and then submit MR job with this configuration to the first
cluster, and after the job is finished, I will submit the second job to the
second cluster with almost same Configuration exception I changed the
property: yarn.resourcemanager.address pointing to the second cluster's RM.
My guess the job.xml of the second job holds all property values of the
first cluster(such as yarn.resourcemanager.scheduler.address) and will
override these properties specified in the second cluster(yarn-site.xml for
example), therefore it will talk to the wrong RM when NM is launching the
container.
Please comment.
BTW, I just tweak the standalone app so that it will load the second
cluster's configuration(yarn-site.xml) before submit the second job, it
seems working.
Thanks
On Thu, Feb 13, 2014 at 4:28 PM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Hi Anfernee,
>
> It sounds most likely that config somehow corrupts. So you have two sets
> of config to start two YARN cluster separately, don't you? If you provide
> more detail about how you config the two clusters, it's easy for the
> community to understand your problem.
>
> - Zhijie
>
>
> On Thu, Feb 13, 2014 at 11:34 AM, Anfernee Xu <an...@gmail.com>wrote:
>
>> I'm at Yarn 2.2.0 release, I configured 2 single-node clusters on my
>> laptop(just for POC and all port conflicts are resolved, and I can see NM
>> and RM is up, webUI shows everything is fine) and I also have a standalone
>> java application. The java application is a kind of job client, it will
>> submit job1 to Cluser #1, once the job is finished, it will submit another
>> job2 to Cluster #2.
>>
>> What I'm seeing is the job1 is doing fine, but job2 failed, I looked
>> source code, and found the NM in cluser2 was talking to cluser1's RM via
>> wrong yarn.resourcemanager.scheduler.address. How that happens? I just want
>> to make sure there's no such issue in real deployment.
>>
>> --
>> --Anfernee
>>
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
--Anfernee
Re: job submission between 2 YARN clusters
Posted by Anfernee Xu <an...@gmail.com>.
Hi Zhijie,
I agree, what I'm doing in the standalone app is that the app loads the
first cluster Configuration(mapred-site.xml, yarn-site.xml) as its default
configuration, and then submit MR job with this configuration to the first
cluster, and after the job is finished, I will submit the second job to the
second cluster with almost same Configuration exception I changed the
property: yarn.resourcemanager.address pointing to the second cluster's RM.
My guess the job.xml of the second job holds all property values of the
first cluster(such as yarn.resourcemanager.scheduler.address) and will
override these properties specified in the second cluster(yarn-site.xml for
example), therefore it will talk to the wrong RM when NM is launching the
container.
Please comment.
BTW, I just tweak the standalone app so that it will load the second
cluster's configuration(yarn-site.xml) before submit the second job, it
seems working.
Thanks
On Thu, Feb 13, 2014 at 4:28 PM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Hi Anfernee,
>
> It sounds most likely that config somehow corrupts. So you have two sets
> of config to start two YARN cluster separately, don't you? If you provide
> more detail about how you config the two clusters, it's easy for the
> community to understand your problem.
>
> - Zhijie
>
>
> On Thu, Feb 13, 2014 at 11:34 AM, Anfernee Xu <an...@gmail.com>wrote:
>
>> I'm at Yarn 2.2.0 release, I configured 2 single-node clusters on my
>> laptop(just for POC and all port conflicts are resolved, and I can see NM
>> and RM is up, webUI shows everything is fine) and I also have a standalone
>> java application. The java application is a kind of job client, it will
>> submit job1 to Cluser #1, once the job is finished, it will submit another
>> job2 to Cluster #2.
>>
>> What I'm seeing is the job1 is doing fine, but job2 failed, I looked
>> source code, and found the NM in cluser2 was talking to cluser1's RM via
>> wrong yarn.resourcemanager.scheduler.address. How that happens? I just want
>> to make sure there's no such issue in real deployment.
>>
>> --
>> --Anfernee
>>
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
--Anfernee
Re: job submission between 2 YARN clusters
Posted by Zhijie Shen <zs...@hortonworks.com>.
Hi Anfernee,
It sounds most likely that config somehow corrupts. So you have two sets of
config to start two YARN cluster separately, don't you? If you provide more
detail about how you config the two clusters, it's easy for the community
to understand your problem.
- Zhijie
On Thu, Feb 13, 2014 at 11:34 AM, Anfernee Xu <an...@gmail.com> wrote:
> I'm at Yarn 2.2.0 release, I configured 2 single-node clusters on my
> laptop(just for POC and all port conflicts are resolved, and I can see NM
> and RM is up, webUI shows everything is fine) and I also have a standalone
> java application. The java application is a kind of job client, it will
> submit job1 to Cluser #1, once the job is finished, it will submit another
> job2 to Cluster #2.
>
> What I'm seeing is the job1 is doing fine, but job2 failed, I looked
> source code, and found the NM in cluser2 was talking to cluser1's RM via
> wrong yarn.resourcemanager.scheduler.address. How that happens? I just want
> to make sure there's no such issue in real deployment.
>
> --
> --Anfernee
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: job submission between 2 YARN clusters
Posted by Zhijie Shen <zs...@hortonworks.com>.
Hi Anfernee,
It sounds most likely that config somehow corrupts. So you have two sets of
config to start two YARN cluster separately, don't you? If you provide more
detail about how you config the two clusters, it's easy for the community
to understand your problem.
- Zhijie
On Thu, Feb 13, 2014 at 11:34 AM, Anfernee Xu <an...@gmail.com> wrote:
> I'm at Yarn 2.2.0 release, I configured 2 single-node clusters on my
> laptop(just for POC and all port conflicts are resolved, and I can see NM
> and RM is up, webUI shows everything is fine) and I also have a standalone
> java application. The java application is a kind of job client, it will
> submit job1 to Cluser #1, once the job is finished, it will submit another
> job2 to Cluster #2.
>
> What I'm seeing is the job1 is doing fine, but job2 failed, I looked
> source code, and found the NM in cluser2 was talking to cluser1's RM via
> wrong yarn.resourcemanager.scheduler.address. How that happens? I just want
> to make sure there's no such issue in real deployment.
>
> --
> --Anfernee
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: job submission between 2 YARN clusters
Posted by Zhijie Shen <zs...@hortonworks.com>.
Hi Anfernee,
It sounds most likely that config somehow corrupts. So you have two sets of
config to start two YARN cluster separately, don't you? If you provide more
detail about how you config the two clusters, it's easy for the community
to understand your problem.
- Zhijie
On Thu, Feb 13, 2014 at 11:34 AM, Anfernee Xu <an...@gmail.com> wrote:
> I'm at Yarn 2.2.0 release, I configured 2 single-node clusters on my
> laptop(just for POC and all port conflicts are resolved, and I can see NM
> and RM is up, webUI shows everything is fine) and I also have a standalone
> java application. The java application is a kind of job client, it will
> submit job1 to Cluser #1, once the job is finished, it will submit another
> job2 to Cluster #2.
>
> What I'm seeing is the job1 is doing fine, but job2 failed, I looked
> source code, and found the NM in cluser2 was talking to cluser1's RM via
> wrong yarn.resourcemanager.scheduler.address. How that happens? I just want
> to make sure there's no such issue in real deployment.
>
> --
> --Anfernee
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: job submission between 2 YARN clusters
Posted by Zhijie Shen <zs...@hortonworks.com>.
Hi Anfernee,
It sounds most likely that config somehow corrupts. So you have two sets of
config to start two YARN cluster separately, don't you? If you provide more
detail about how you config the two clusters, it's easy for the community
to understand your problem.
- Zhijie
On Thu, Feb 13, 2014 at 11:34 AM, Anfernee Xu <an...@gmail.com> wrote:
> I'm at Yarn 2.2.0 release, I configured 2 single-node clusters on my
> laptop(just for POC and all port conflicts are resolved, and I can see NM
> and RM is up, webUI shows everything is fine) and I also have a standalone
> java application. The java application is a kind of job client, it will
> submit job1 to Cluser #1, once the job is finished, it will submit another
> job2 to Cluster #2.
>
> What I'm seeing is the job1 is doing fine, but job2 failed, I looked
> source code, and found the NM in cluser2 was talking to cluser1's RM via
> wrong yarn.resourcemanager.scheduler.address. How that happens? I just want
> to make sure there's no such issue in real deployment.
>
> --
> --Anfernee
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.