You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Grandl Robert <rg...@yahoo.com> on 2013/11/25 19:38:18 UTC
Fw: blacklist additions/removals not propagated to RM
I am not sure I posted this question in the right thread.
Do you have some thoughts with the below mail ? Is there any configuration parameter for RM to enable blacklisting ? Anyway, it seems my blacklist is sent empty to RM, so I think is an AM problem though.
Please let me know about your thoughts,
robert
On Saturday, November 23, 2013 2:31 PM, Grandl Robert <rg...@yahoo.com> wrote:
Hi,
I am not 100% sure it's a bug in AM, but it seems so. I am blacklisting some hosts, which are send through an AllocateRequest in makeRemoteRequest method in RMContainerRequest.java.
However, the blacklist is empty at RM. I traced all over the places where this is lost, and I found that in ApplicationMasterProtocolPBClientImpl.java, in allocate(AllocateRequest request) method, if I print the request.getResourceBlacklistRequest().getBlacklistAdditiions() I see the list, but after it does AllocateRequestProto requestProto = (..).getProto(),
and I print the requestProto.getBlacklistRequest().getBlacklistAdditions() now is EMPTY.
So basically, AM send an empty black list to RM. I went even further down and traced till YarnServiceProtos.java(in hadoop-yarn-api), but at this point all the logs I have, are removed. it seems this code is regenerated every time I am compiling yarn-api ....
Is there something I am missing guys, or is a bug ? Interesting is that release containers list is propagated correctly but I think it comes on other path. I am running hadoop-yarn 2.3.
Thanks,
robert
RE: blacklist additions/removals not propagated to RM
Posted by Bikas Saha <bi...@hortonworks.com>.
Please open a jira for the issue with the details you describe.
Thanks
Bikas
-----Original Message-----
From: Grandl Robert [mailto:rgrandl@yahoo.com]
Sent: Monday, November 25, 2013 10:38 AM
To: yarn-dev@hadoop.apache.org
Subject: Fw: blacklist additions/removals not propagated to RM
I am not sure I posted this question in the right thread.
Do you have some thoughts with the below mail ? Is there any configuration
parameter for RM to enable blacklisting ? Anyway, it seems my blacklist is
sent empty to RM, so I think is an AM problem though.
Please let me know about your thoughts,
robert
On Saturday, November 23, 2013 2:31 PM, Grandl Robert <rg...@yahoo.com>
wrote:
Hi,
I am not 100% sure it's a bug in AM, but it seems so. I am blacklisting
some hosts, which are send through an AllocateRequest in makeRemoteRequest
method in RMContainerRequest.java.
However, the blacklist is empty at RM. I traced all over the places where
this is lost, and I found that in
ApplicationMasterProtocolPBClientImpl.java, in allocate(AllocateRequest
request) method, if I print the
request.getResourceBlacklistRequest().getBlacklistAdditiions() I see the
list, but after it does AllocateRequestProto requestProto =
(..).getProto(), and I print the
requestProto.getBlacklistRequest().getBlacklistAdditions() now is EMPTY.
So basically, AM send an empty black list to RM. I went even further down
and traced till YarnServiceProtos.java(in hadoop-yarn-api), but at this
point all the logs I have, are removed. it seems this code is regenerated
every time I am compiling yarn-api ....
Is there something I am missing guys, or is a bug ? Interesting is that
release containers list is propagated correctly but I think it comes on
other path. I am running hadoop-yarn 2.3.
Thanks,
robert
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Fw: blacklist additions/removals not propagated to RM
Posted by Zhijie Shen <zs...@hortonworks.com>.
BTW, if anybody can help commit YARN-1301, which helps debugging the
blacklist issues. Thanks! - Zhijie
On Mon, Nov 25, 2013 at 11:08 AM, Zhijie Shen <zs...@hortonworks.com> wrote:
> Hi Grandl,
>
> Have you checked the following log message "Update the blacklist for ..."?
> What did it tell?
>
> BTW, YarnScheduler#allocate takes the blacklist additions/removes. You may
> also want to verify YarnScheduler#allocate gets the empty list or not.
>
> Thanks,
> Zhijie
>
>
> On Mon, Nov 25, 2013 at 10:38 AM, Grandl Robert <rg...@yahoo.com> wrote:
>
>> I am not sure I posted this question in the right thread.
>>
>>
>> Do you have some thoughts with the below mail ? Is there any
>> configuration parameter for RM to enable blacklisting ? Anyway, it seems my
>> blacklist is sent empty to RM, so I think is an AM problem though.
>>
>>
>> Please let me know about your thoughts,
>>
>> robert
>>
>>
>>
>>
>> On Saturday, November 23, 2013 2:31 PM, Grandl Robert <rg...@yahoo.com>
>> wrote:
>>
>> Hi,
>>
>> I am not 100% sure it's a bug in AM, but it seems so. I am blacklisting
>> some hosts, which are send through an AllocateRequest in makeRemoteRequest
>> method in RMContainerRequest.java.
>>
>> However, the blacklist is empty at RM. I traced all over the places where
>> this is lost, and I found that in
>> ApplicationMasterProtocolPBClientImpl.java, in allocate(AllocateRequest
>> request) method, if I print the
>> request.getResourceBlacklistRequest().getBlacklistAdditiions() I see the
>> list, but after it does AllocateRequestProto requestProto = (..).getProto(),
>> and I print the
>> requestProto.getBlacklistRequest().getBlacklistAdditions() now is EMPTY.
>>
>> So basically, AM send an empty black list to RM. I went even further down
>> and traced till YarnServiceProtos.java(in hadoop-yarn-api), but at this
>> point all the logs I have, are removed. it seems this code is regenerated
>> every time I am compiling yarn-api ....
>>
>> Is there something I am missing guys, or is a bug ? Interesting is that
>> release containers list is propagated correctly but I think it comes on
>> other path. I am running hadoop-yarn 2.3.
>>
>> Thanks,
>> robert
>>
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Fw: blacklist additions/removals not propagated to RM
Posted by Grandl Robert <rg...@yahoo.com>.
I checked those places Zhijie. As I mentioned before in ApplicationMasterProtocolPBClientImpl#allocate, which is further down in YarnScheduler#allocate, if I print the blacklist at the beginning of the function:
LOG.info("RM --> allocate request from AM; blacklist_a="+request.getResourceBlacklistRequest().getBlacklistAdditions().toString());
it shows me the right list, but if I print after requestProto:
LOG.info("RM --> allocate request from AM; blacklist_a="+requestProto.getBlacklistRequest().getBlacklistAdditionsList().toString());
it is empty.
It seems things are screwed up somewhere in yarn-api. I will open a jira and maybe you can take a look please.
Thanks,
robert
On Monday, November 25, 2013 11:09 AM, Zhijie Shen <zs...@hortonworks.com> wrote:
Hi Grandl,
Have you checked the following log message "Update the blacklist for ..."?
What did it tell?
BTW, YarnScheduler#allocate takes the blacklist additions/removes. You may
also want to verify YarnScheduler#allocate gets the empty list or not.
Thanks,
Zhijie
On Mon, Nov 25, 2013 at 10:38 AM, Grandl Robert <rg...@yahoo.com> wrote:
> I am not sure I posted this question in the right thread.
>
>
> Do you have some thoughts with the below mail ? Is there any configuration
> parameter for RM to enable blacklisting ? Anyway, it seems my blacklist is
> sent empty to RM, so I think is an AM problem though.
>
>
> Please let me know about your thoughts,
>
> robert
>
>
>
>
> On Saturday, November 23, 2013 2:31 PM, Grandl Robert <rg...@yahoo.com>
> wrote:
>
> Hi,
>
> I am not 100% sure it's a bug in AM, but it seems so. I am blacklisting
> some hosts, which are send through an AllocateRequest in makeRemoteRequest
> method in RMContainerRequest.java.
>
> However, the blacklist is empty at RM. I traced all over the places where
> this is lost, and I found that in
> ApplicationMasterProtocolPBClientImpl.java, in allocate(AllocateRequest
> request) method, if I print the
> request.getResourceBlacklistRequest().getBlacklistAdditiions() I see the
> list, but after it does AllocateRequestProto requestProto = (..).getProto(),
> and I print the requestProto.getBlacklistRequest().getBlacklistAdditions()
> now is EMPTY.
>
> So basically, AM send an empty black list to RM. I went even further down
> and traced till YarnServiceProtos.java(in hadoop-yarn-api), but at this
> point all the logs I have, are removed. it seems this code is regenerated
> every time I am compiling yarn-api ....
>
> Is there something I am missing guys, or is a bug ? Interesting is that
> release containers list is propagated correctly but I think it comes on
> other path. I am running hadoop-yarn 2.3.
>
> Thanks,
> robert
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Fw: blacklist additions/removals not propagated to RM
Posted by Zhijie Shen <zs...@hortonworks.com>.
Hi Grandl,
Have you checked the following log message "Update the blacklist for ..."?
What did it tell?
BTW, YarnScheduler#allocate takes the blacklist additions/removes. You may
also want to verify YarnScheduler#allocate gets the empty list or not.
Thanks,
Zhijie
On Mon, Nov 25, 2013 at 10:38 AM, Grandl Robert <rg...@yahoo.com> wrote:
> I am not sure I posted this question in the right thread.
>
>
> Do you have some thoughts with the below mail ? Is there any configuration
> parameter for RM to enable blacklisting ? Anyway, it seems my blacklist is
> sent empty to RM, so I think is an AM problem though.
>
>
> Please let me know about your thoughts,
>
> robert
>
>
>
>
> On Saturday, November 23, 2013 2:31 PM, Grandl Robert <rg...@yahoo.com>
> wrote:
>
> Hi,
>
> I am not 100% sure it's a bug in AM, but it seems so. I am blacklisting
> some hosts, which are send through an AllocateRequest in makeRemoteRequest
> method in RMContainerRequest.java.
>
> However, the blacklist is empty at RM. I traced all over the places where
> this is lost, and I found that in
> ApplicationMasterProtocolPBClientImpl.java, in allocate(AllocateRequest
> request) method, if I print the
> request.getResourceBlacklistRequest().getBlacklistAdditiions() I see the
> list, but after it does AllocateRequestProto requestProto = (..).getProto(),
> and I print the requestProto.getBlacklistRequest().getBlacklistAdditions()
> now is EMPTY.
>
> So basically, AM send an empty black list to RM. I went even further down
> and traced till YarnServiceProtos.java(in hadoop-yarn-api), but at this
> point all the logs I have, are removed. it seems this code is regenerated
> every time I am compiling yarn-api ....
>
> Is there something I am missing guys, or is a bug ? Interesting is that
> release containers list is propagated correctly but I think it comes on
> other path. I am running hadoop-yarn 2.3.
>
> Thanks,
> robert
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.