You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Krishna Kishore Bonagiri <wr...@gmail.com> on 2013/07/08 15:48:24 UTC
whitelist feature of YARN
Hi,
Can someone please point to some example code of how to use the whitelist
feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta and want to
use this feature.
It would be great if you can point me to some description of what this
white listing feature is, I have gone through some JIRA logs related to
this but more concrete explanation would be helpful.
Thanks,
Kishore
Re: whitelist feature of YARN
Posted by Arun C Murthy <ac...@hortonworks.com>.
Go ahead - it's very close, I'll probably create an rc2 in the next couple of days.
On Aug 7, 2013, at 2:06 AM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
> Sandy,
> Thanks again. I found RC1 for 2.1.0-beta available at http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc1/
> Would this have the fix for YARN-521? and, can I use that?
>
> -Kishore
>
>
> On Wed, Aug 7, 2013 at 12:35 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Responses inline:
>
>
> On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
> Hi Sandy,
>
> Thanks for the reply and it is good to know YARN-521 is done! Please answer my following questions
>
> 1) when is 2.1.0-beta going to be released? is it soon or do you suggest me take it from the trunk or is there a recent release candidate available?
>
> We're very close and my guess would be no later than the end of the month (don't hold me to this).
>
> 2) I have recently changed my application to use the new Asynchronous interfaces. I am hoping it works with that too, correct me if I am wrong.
> ContainerRequest is shared by the async interfaces as well so it should work here.
>
> 3) Change in interface:
>
> The old interface for ContainerRequest constructor used to be this:
>
> public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority, int containerCount);
>
> where as now it is changed to
>
> a) public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority)
> &
>
> b) public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority, boolean relaxLocality)
>
> that means the old argument containerCount is gone! How would I be able to specify how many containers do I need?
>
> We now expect that you submit a ContainerRequest for each container you want.
>
> -Kishore
>
>
>
>
> On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> YARN-521, which brings whitelisting to the AMRMClient APIs, is now included in 2.1.0-beta. Check out the doc for the relaxLocality paramater in ContainerRequest in AMRMClient: https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and I can help clarify here if anything's confusing.
>
> -Sandy
>
>
> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
> Hi Sandy,
>
> Yes, I have been using AMRMClient APIs. I am planning to shift to whatever way is this white list feature is supported with. But am not sure what is meant by submitting ResourceRequests directly to RM. Can you please elaborate on this or give me a pointer to some example code on how to do it...
>
> Thanks for the reply,
>
> -Kishore
>
>
> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Krishna,
>
> From your previous email, it looks like you are using the AMRMClient APIs. Support for whitelisting is not yet supported through them. I am working on this in YARN-521, which should be included in the next release after 2.1.0-beta. If you are submitting ResourceRequests directly to the RM, you can whitelist a node by
> * setting the relaxLocality flag on the node-level ResourceRequest to true
> * setting the relaxLocality flag on the corresponding rack-level ResourceRequest to false
> * setting the relaxLocality flag on the corresponding any-level ResourceRequest to false
>
> -Sandy
>
>
> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
> Hi,
>
> Can someone please point to some example code of how to use the whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta and want to use this feature.
>
> It would be great if you can point me to some description of what this white listing feature is, I have gone through some JIRA logs related to this but more concrete explanation would be helpful.
>
> Thanks,
> Kishore
>
>
>
>
>
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: whitelist feature of YARN
Posted by Arun C Murthy <ac...@hortonworks.com>.
Go ahead - it's very close, I'll probably create an rc2 in the next couple of days.
On Aug 7, 2013, at 2:06 AM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
> Sandy,
> Thanks again. I found RC1 for 2.1.0-beta available at http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc1/
> Would this have the fix for YARN-521? and, can I use that?
>
> -Kishore
>
>
> On Wed, Aug 7, 2013 at 12:35 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Responses inline:
>
>
> On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
> Hi Sandy,
>
> Thanks for the reply and it is good to know YARN-521 is done! Please answer my following questions
>
> 1) when is 2.1.0-beta going to be released? is it soon or do you suggest me take it from the trunk or is there a recent release candidate available?
>
> We're very close and my guess would be no later than the end of the month (don't hold me to this).
>
> 2) I have recently changed my application to use the new Asynchronous interfaces. I am hoping it works with that too, correct me if I am wrong.
> ContainerRequest is shared by the async interfaces as well so it should work here.
>
> 3) Change in interface:
>
> The old interface for ContainerRequest constructor used to be this:
>
> public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority, int containerCount);
>
> where as now it is changed to
>
> a) public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority)
> &
>
> b) public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority, boolean relaxLocality)
>
> that means the old argument containerCount is gone! How would I be able to specify how many containers do I need?
>
> We now expect that you submit a ContainerRequest for each container you want.
>
> -Kishore
>
>
>
>
> On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> YARN-521, which brings whitelisting to the AMRMClient APIs, is now included in 2.1.0-beta. Check out the doc for the relaxLocality paramater in ContainerRequest in AMRMClient: https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and I can help clarify here if anything's confusing.
>
> -Sandy
>
>
> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
> Hi Sandy,
>
> Yes, I have been using AMRMClient APIs. I am planning to shift to whatever way is this white list feature is supported with. But am not sure what is meant by submitting ResourceRequests directly to RM. Can you please elaborate on this or give me a pointer to some example code on how to do it...
>
> Thanks for the reply,
>
> -Kishore
>
>
> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Krishna,
>
> From your previous email, it looks like you are using the AMRMClient APIs. Support for whitelisting is not yet supported through them. I am working on this in YARN-521, which should be included in the next release after 2.1.0-beta. If you are submitting ResourceRequests directly to the RM, you can whitelist a node by
> * setting the relaxLocality flag on the node-level ResourceRequest to true
> * setting the relaxLocality flag on the corresponding rack-level ResourceRequest to false
> * setting the relaxLocality flag on the corresponding any-level ResourceRequest to false
>
> -Sandy
>
>
> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
> Hi,
>
> Can someone please point to some example code of how to use the whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta and want to use this feature.
>
> It would be great if you can point me to some description of what this white listing feature is, I have gone through some JIRA logs related to this but more concrete explanation would be helpful.
>
> Thanks,
> Kishore
>
>
>
>
>
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: whitelist feature of YARN
Posted by Arun C Murthy <ac...@hortonworks.com>.
Go ahead - it's very close, I'll probably create an rc2 in the next couple of days.
On Aug 7, 2013, at 2:06 AM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
> Sandy,
> Thanks again. I found RC1 for 2.1.0-beta available at http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc1/
> Would this have the fix for YARN-521? and, can I use that?
>
> -Kishore
>
>
> On Wed, Aug 7, 2013 at 12:35 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Responses inline:
>
>
> On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
> Hi Sandy,
>
> Thanks for the reply and it is good to know YARN-521 is done! Please answer my following questions
>
> 1) when is 2.1.0-beta going to be released? is it soon or do you suggest me take it from the trunk or is there a recent release candidate available?
>
> We're very close and my guess would be no later than the end of the month (don't hold me to this).
>
> 2) I have recently changed my application to use the new Asynchronous interfaces. I am hoping it works with that too, correct me if I am wrong.
> ContainerRequest is shared by the async interfaces as well so it should work here.
>
> 3) Change in interface:
>
> The old interface for ContainerRequest constructor used to be this:
>
> public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority, int containerCount);
>
> where as now it is changed to
>
> a) public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority)
> &
>
> b) public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority, boolean relaxLocality)
>
> that means the old argument containerCount is gone! How would I be able to specify how many containers do I need?
>
> We now expect that you submit a ContainerRequest for each container you want.
>
> -Kishore
>
>
>
>
> On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> YARN-521, which brings whitelisting to the AMRMClient APIs, is now included in 2.1.0-beta. Check out the doc for the relaxLocality paramater in ContainerRequest in AMRMClient: https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and I can help clarify here if anything's confusing.
>
> -Sandy
>
>
> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
> Hi Sandy,
>
> Yes, I have been using AMRMClient APIs. I am planning to shift to whatever way is this white list feature is supported with. But am not sure what is meant by submitting ResourceRequests directly to RM. Can you please elaborate on this or give me a pointer to some example code on how to do it...
>
> Thanks for the reply,
>
> -Kishore
>
>
> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Krishna,
>
> From your previous email, it looks like you are using the AMRMClient APIs. Support for whitelisting is not yet supported through them. I am working on this in YARN-521, which should be included in the next release after 2.1.0-beta. If you are submitting ResourceRequests directly to the RM, you can whitelist a node by
> * setting the relaxLocality flag on the node-level ResourceRequest to true
> * setting the relaxLocality flag on the corresponding rack-level ResourceRequest to false
> * setting the relaxLocality flag on the corresponding any-level ResourceRequest to false
>
> -Sandy
>
>
> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
> Hi,
>
> Can someone please point to some example code of how to use the whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta and want to use this feature.
>
> It would be great if you can point me to some description of what this white listing feature is, I have gone through some JIRA logs related to this but more concrete explanation would be helpful.
>
> Thanks,
> Kishore
>
>
>
>
>
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: whitelist feature of YARN
Posted by Arun C Murthy <ac...@hortonworks.com>.
Go ahead - it's very close, I'll probably create an rc2 in the next couple of days.
On Aug 7, 2013, at 2:06 AM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
> Sandy,
> Thanks again. I found RC1 for 2.1.0-beta available at http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc1/
> Would this have the fix for YARN-521? and, can I use that?
>
> -Kishore
>
>
> On Wed, Aug 7, 2013 at 12:35 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Responses inline:
>
>
> On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
> Hi Sandy,
>
> Thanks for the reply and it is good to know YARN-521 is done! Please answer my following questions
>
> 1) when is 2.1.0-beta going to be released? is it soon or do you suggest me take it from the trunk or is there a recent release candidate available?
>
> We're very close and my guess would be no later than the end of the month (don't hold me to this).
>
> 2) I have recently changed my application to use the new Asynchronous interfaces. I am hoping it works with that too, correct me if I am wrong.
> ContainerRequest is shared by the async interfaces as well so it should work here.
>
> 3) Change in interface:
>
> The old interface for ContainerRequest constructor used to be this:
>
> public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority, int containerCount);
>
> where as now it is changed to
>
> a) public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority)
> &
>
> b) public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority, boolean relaxLocality)
>
> that means the old argument containerCount is gone! How would I be able to specify how many containers do I need?
>
> We now expect that you submit a ContainerRequest for each container you want.
>
> -Kishore
>
>
>
>
> On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> YARN-521, which brings whitelisting to the AMRMClient APIs, is now included in 2.1.0-beta. Check out the doc for the relaxLocality paramater in ContainerRequest in AMRMClient: https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and I can help clarify here if anything's confusing.
>
> -Sandy
>
>
> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
> Hi Sandy,
>
> Yes, I have been using AMRMClient APIs. I am planning to shift to whatever way is this white list feature is supported with. But am not sure what is meant by submitting ResourceRequests directly to RM. Can you please elaborate on this or give me a pointer to some example code on how to do it...
>
> Thanks for the reply,
>
> -Kishore
>
>
> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Krishna,
>
> From your previous email, it looks like you are using the AMRMClient APIs. Support for whitelisting is not yet supported through them. I am working on this in YARN-521, which should be included in the next release after 2.1.0-beta. If you are submitting ResourceRequests directly to the RM, you can whitelist a node by
> * setting the relaxLocality flag on the node-level ResourceRequest to true
> * setting the relaxLocality flag on the corresponding rack-level ResourceRequest to false
> * setting the relaxLocality flag on the corresponding any-level ResourceRequest to false
>
> -Sandy
>
>
> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <wr...@gmail.com> wrote:
> Hi,
>
> Can someone please point to some example code of how to use the whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta and want to use this feature.
>
> It would be great if you can point me to some description of what this white listing feature is, I have gone through some JIRA logs related to this but more concrete explanation would be helpful.
>
> Thanks,
> Kishore
>
>
>
>
>
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: whitelist feature of YARN
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Sandy,
Thanks again. I found RC1 for 2.1.0-beta available at
http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc1/
Would this have the fix for YARN-521? and, can I use that?
-Kishore
On Wed, Aug 7, 2013 at 12:35 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Responses inline:
>
>
> On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Sandy,
>>
>> Thanks for the reply and it is good to know YARN-521 is done! Please
>> answer my following questions
>>
>> 1) when is 2.1.0-beta going to be released? is it soon or do you suggest
>> me take it from the trunk or is there a recent release candidate available?
>>
>> We're very close and my guess would be no later than the end of the
> month (don't hold me to this).
>
>
>> 2) I have recently changed my application to use the new Asynchronous
>> interfaces. I am hoping it works with that too, correct me if I am wrong.
>>
> ContainerRequest is shared by the async interfaces as well so it should
> work here.
>
>>
>> 3) Change in interface:
>>
>> The old interface for ContainerRequest constructor used to be this:
>>
>> public ContainerRequest(Resource capability, String[] nodes,
>> String[] racks, Priority priority, int containerCount);
>>
>> where as now it is changed to
>>
>> a) public ContainerRequest(Resource capability, String[] nodes,
>> String[] racks, Priority priority)
>> &
>>
>> b) public ContainerRequest(Resource capability, String[] nodes,
>> String[] racks, Priority priority, boolean relaxLocality)
>>
>> that means the old argument containerCount is gone! How would I be able
>> to specify how many containers do I need?
>>
>> We now expect that you submit a ContainerRequest for each container you
> want.
>
>
>> -Kishore
>>
>>
>>
>>
>> On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>>
>>> YARN-521, which brings whitelisting to the AMRMClient APIs, is now
>>> included in 2.1.0-beta. Check out the doc for the relaxLocality paramater
>>> in ContainerRequest in AMRMClient:
>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and
>>> I can help clarify here if anything's confusing.
>>>
>>> -Sandy
>>>
>>>
>>> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi Sandy,
>>>>
>>>> Yes, I have been using AMRMClient APIs. I am planning to shift to
>>>> whatever way is this white list feature is supported with. But am not sure
>>>> what is meant by submitting ResourceRequests directly to RM. Can you please
>>>> elaborate on this or give me a pointer to some example code on how to do
>>>> it...
>>>>
>>>> Thanks for the reply,
>>>>
>>>> -Kishore
>>>>
>>>>
>>>> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com>wrote:
>>>>
>>>>> Hi Krishna,
>>>>>
>>>>> From your previous email, it looks like you are using the AMRMClient
>>>>> APIs. Support for whitelisting is not yet supported through them. I am
>>>>> working on this in YARN-521, which should be included in the next release
>>>>> after 2.1.0-beta. If you are submitting ResourceRequests directly to the
>>>>> RM, you can whitelist a node by
>>>>> * setting the relaxLocality flag on the node-level ResourceRequest to
>>>>> true
>>>>> * setting the relaxLocality flag on the corresponding rack-level
>>>>> ResourceRequest to false
>>>>> * setting the relaxLocality flag on the corresponding any-level
>>>>> ResourceRequest to false
>>>>>
>>>>> -Sandy
>>>>>
>>>>>
>>>>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>>>>> write2kishore@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Can someone please point to some example code of how to use the
>>>>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>>>>> and want to use this feature.
>>>>>>
>>>>>> It would be great if you can point me to some description of what
>>>>>> this white listing feature is, I have gone through some JIRA logs related
>>>>>> to this but more concrete explanation would be helpful.
>>>>>>
>>>>>> Thanks,
>>>>>> Kishore
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: whitelist feature of YARN
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Sandy,
Thanks again. I found RC1 for 2.1.0-beta available at
http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc1/
Would this have the fix for YARN-521? and, can I use that?
-Kishore
On Wed, Aug 7, 2013 at 12:35 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Responses inline:
>
>
> On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Sandy,
>>
>> Thanks for the reply and it is good to know YARN-521 is done! Please
>> answer my following questions
>>
>> 1) when is 2.1.0-beta going to be released? is it soon or do you suggest
>> me take it from the trunk or is there a recent release candidate available?
>>
>> We're very close and my guess would be no later than the end of the
> month (don't hold me to this).
>
>
>> 2) I have recently changed my application to use the new Asynchronous
>> interfaces. I am hoping it works with that too, correct me if I am wrong.
>>
> ContainerRequest is shared by the async interfaces as well so it should
> work here.
>
>>
>> 3) Change in interface:
>>
>> The old interface for ContainerRequest constructor used to be this:
>>
>> public ContainerRequest(Resource capability, String[] nodes,
>> String[] racks, Priority priority, int containerCount);
>>
>> where as now it is changed to
>>
>> a) public ContainerRequest(Resource capability, String[] nodes,
>> String[] racks, Priority priority)
>> &
>>
>> b) public ContainerRequest(Resource capability, String[] nodes,
>> String[] racks, Priority priority, boolean relaxLocality)
>>
>> that means the old argument containerCount is gone! How would I be able
>> to specify how many containers do I need?
>>
>> We now expect that you submit a ContainerRequest for each container you
> want.
>
>
>> -Kishore
>>
>>
>>
>>
>> On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>>
>>> YARN-521, which brings whitelisting to the AMRMClient APIs, is now
>>> included in 2.1.0-beta. Check out the doc for the relaxLocality paramater
>>> in ContainerRequest in AMRMClient:
>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and
>>> I can help clarify here if anything's confusing.
>>>
>>> -Sandy
>>>
>>>
>>> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi Sandy,
>>>>
>>>> Yes, I have been using AMRMClient APIs. I am planning to shift to
>>>> whatever way is this white list feature is supported with. But am not sure
>>>> what is meant by submitting ResourceRequests directly to RM. Can you please
>>>> elaborate on this or give me a pointer to some example code on how to do
>>>> it...
>>>>
>>>> Thanks for the reply,
>>>>
>>>> -Kishore
>>>>
>>>>
>>>> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com>wrote:
>>>>
>>>>> Hi Krishna,
>>>>>
>>>>> From your previous email, it looks like you are using the AMRMClient
>>>>> APIs. Support for whitelisting is not yet supported through them. I am
>>>>> working on this in YARN-521, which should be included in the next release
>>>>> after 2.1.0-beta. If you are submitting ResourceRequests directly to the
>>>>> RM, you can whitelist a node by
>>>>> * setting the relaxLocality flag on the node-level ResourceRequest to
>>>>> true
>>>>> * setting the relaxLocality flag on the corresponding rack-level
>>>>> ResourceRequest to false
>>>>> * setting the relaxLocality flag on the corresponding any-level
>>>>> ResourceRequest to false
>>>>>
>>>>> -Sandy
>>>>>
>>>>>
>>>>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>>>>> write2kishore@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Can someone please point to some example code of how to use the
>>>>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>>>>> and want to use this feature.
>>>>>>
>>>>>> It would be great if you can point me to some description of what
>>>>>> this white listing feature is, I have gone through some JIRA logs related
>>>>>> to this but more concrete explanation would be helpful.
>>>>>>
>>>>>> Thanks,
>>>>>> Kishore
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: whitelist feature of YARN
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Sandy,
Thanks again. I found RC1 for 2.1.0-beta available at
http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc1/
Would this have the fix for YARN-521? and, can I use that?
-Kishore
On Wed, Aug 7, 2013 at 12:35 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Responses inline:
>
>
> On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Sandy,
>>
>> Thanks for the reply and it is good to know YARN-521 is done! Please
>> answer my following questions
>>
>> 1) when is 2.1.0-beta going to be released? is it soon or do you suggest
>> me take it from the trunk or is there a recent release candidate available?
>>
>> We're very close and my guess would be no later than the end of the
> month (don't hold me to this).
>
>
>> 2) I have recently changed my application to use the new Asynchronous
>> interfaces. I am hoping it works with that too, correct me if I am wrong.
>>
> ContainerRequest is shared by the async interfaces as well so it should
> work here.
>
>>
>> 3) Change in interface:
>>
>> The old interface for ContainerRequest constructor used to be this:
>>
>> public ContainerRequest(Resource capability, String[] nodes,
>> String[] racks, Priority priority, int containerCount);
>>
>> where as now it is changed to
>>
>> a) public ContainerRequest(Resource capability, String[] nodes,
>> String[] racks, Priority priority)
>> &
>>
>> b) public ContainerRequest(Resource capability, String[] nodes,
>> String[] racks, Priority priority, boolean relaxLocality)
>>
>> that means the old argument containerCount is gone! How would I be able
>> to specify how many containers do I need?
>>
>> We now expect that you submit a ContainerRequest for each container you
> want.
>
>
>> -Kishore
>>
>>
>>
>>
>> On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>>
>>> YARN-521, which brings whitelisting to the AMRMClient APIs, is now
>>> included in 2.1.0-beta. Check out the doc for the relaxLocality paramater
>>> in ContainerRequest in AMRMClient:
>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and
>>> I can help clarify here if anything's confusing.
>>>
>>> -Sandy
>>>
>>>
>>> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi Sandy,
>>>>
>>>> Yes, I have been using AMRMClient APIs. I am planning to shift to
>>>> whatever way is this white list feature is supported with. But am not sure
>>>> what is meant by submitting ResourceRequests directly to RM. Can you please
>>>> elaborate on this or give me a pointer to some example code on how to do
>>>> it...
>>>>
>>>> Thanks for the reply,
>>>>
>>>> -Kishore
>>>>
>>>>
>>>> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com>wrote:
>>>>
>>>>> Hi Krishna,
>>>>>
>>>>> From your previous email, it looks like you are using the AMRMClient
>>>>> APIs. Support for whitelisting is not yet supported through them. I am
>>>>> working on this in YARN-521, which should be included in the next release
>>>>> after 2.1.0-beta. If you are submitting ResourceRequests directly to the
>>>>> RM, you can whitelist a node by
>>>>> * setting the relaxLocality flag on the node-level ResourceRequest to
>>>>> true
>>>>> * setting the relaxLocality flag on the corresponding rack-level
>>>>> ResourceRequest to false
>>>>> * setting the relaxLocality flag on the corresponding any-level
>>>>> ResourceRequest to false
>>>>>
>>>>> -Sandy
>>>>>
>>>>>
>>>>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>>>>> write2kishore@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Can someone please point to some example code of how to use the
>>>>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>>>>> and want to use this feature.
>>>>>>
>>>>>> It would be great if you can point me to some description of what
>>>>>> this white listing feature is, I have gone through some JIRA logs related
>>>>>> to this but more concrete explanation would be helpful.
>>>>>>
>>>>>> Thanks,
>>>>>> Kishore
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: whitelist feature of YARN
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Sandy,
Thanks again. I found RC1 for 2.1.0-beta available at
http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc1/
Would this have the fix for YARN-521? and, can I use that?
-Kishore
On Wed, Aug 7, 2013 at 12:35 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Responses inline:
>
>
> On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Sandy,
>>
>> Thanks for the reply and it is good to know YARN-521 is done! Please
>> answer my following questions
>>
>> 1) when is 2.1.0-beta going to be released? is it soon or do you suggest
>> me take it from the trunk or is there a recent release candidate available?
>>
>> We're very close and my guess would be no later than the end of the
> month (don't hold me to this).
>
>
>> 2) I have recently changed my application to use the new Asynchronous
>> interfaces. I am hoping it works with that too, correct me if I am wrong.
>>
> ContainerRequest is shared by the async interfaces as well so it should
> work here.
>
>>
>> 3) Change in interface:
>>
>> The old interface for ContainerRequest constructor used to be this:
>>
>> public ContainerRequest(Resource capability, String[] nodes,
>> String[] racks, Priority priority, int containerCount);
>>
>> where as now it is changed to
>>
>> a) public ContainerRequest(Resource capability, String[] nodes,
>> String[] racks, Priority priority)
>> &
>>
>> b) public ContainerRequest(Resource capability, String[] nodes,
>> String[] racks, Priority priority, boolean relaxLocality)
>>
>> that means the old argument containerCount is gone! How would I be able
>> to specify how many containers do I need?
>>
>> We now expect that you submit a ContainerRequest for each container you
> want.
>
>
>> -Kishore
>>
>>
>>
>>
>> On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>>
>>> YARN-521, which brings whitelisting to the AMRMClient APIs, is now
>>> included in 2.1.0-beta. Check out the doc for the relaxLocality paramater
>>> in ContainerRequest in AMRMClient:
>>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and
>>> I can help clarify here if anything's confusing.
>>>
>>> -Sandy
>>>
>>>
>>> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi Sandy,
>>>>
>>>> Yes, I have been using AMRMClient APIs. I am planning to shift to
>>>> whatever way is this white list feature is supported with. But am not sure
>>>> what is meant by submitting ResourceRequests directly to RM. Can you please
>>>> elaborate on this or give me a pointer to some example code on how to do
>>>> it...
>>>>
>>>> Thanks for the reply,
>>>>
>>>> -Kishore
>>>>
>>>>
>>>> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com>wrote:
>>>>
>>>>> Hi Krishna,
>>>>>
>>>>> From your previous email, it looks like you are using the AMRMClient
>>>>> APIs. Support for whitelisting is not yet supported through them. I am
>>>>> working on this in YARN-521, which should be included in the next release
>>>>> after 2.1.0-beta. If you are submitting ResourceRequests directly to the
>>>>> RM, you can whitelist a node by
>>>>> * setting the relaxLocality flag on the node-level ResourceRequest to
>>>>> true
>>>>> * setting the relaxLocality flag on the corresponding rack-level
>>>>> ResourceRequest to false
>>>>> * setting the relaxLocality flag on the corresponding any-level
>>>>> ResourceRequest to false
>>>>>
>>>>> -Sandy
>>>>>
>>>>>
>>>>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>>>>> write2kishore@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Can someone please point to some example code of how to use the
>>>>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>>>>> and want to use this feature.
>>>>>>
>>>>>> It would be great if you can point me to some description of what
>>>>>> this white listing feature is, I have gone through some JIRA logs related
>>>>>> to this but more concrete explanation would be helpful.
>>>>>>
>>>>>> Thanks,
>>>>>> Kishore
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: whitelist feature of YARN
Posted by Sandy Ryza <sa...@cloudera.com>.
Responses inline:
On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:
> Hi Sandy,
>
> Thanks for the reply and it is good to know YARN-521 is done! Please
> answer my following questions
>
> 1) when is 2.1.0-beta going to be released? is it soon or do you suggest
> me take it from the trunk or is there a recent release candidate available?
>
> We're very close and my guess would be no later than the end of the
month (don't hold me to this).
> 2) I have recently changed my application to use the new Asynchronous
> interfaces. I am hoping it works with that too, correct me if I am wrong.
>
ContainerRequest is shared by the async interfaces as well so it should
work here.
>
> 3) Change in interface:
>
> The old interface for ContainerRequest constructor used to be this:
>
> public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority, int containerCount);
>
> where as now it is changed to
>
> a) public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority)
> &
>
> b) public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority, boolean relaxLocality)
>
> that means the old argument containerCount is gone! How would I be able to
> specify how many containers do I need?
>
> We now expect that you submit a ContainerRequest for each container you
want.
> -Kishore
>
>
>
>
> On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> YARN-521, which brings whitelisting to the AMRMClient APIs, is now
>> included in 2.1.0-beta. Check out the doc for the relaxLocality paramater
>> in ContainerRequest in AMRMClient:
>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and
>> I can help clarify here if anything's confusing.
>>
>> -Sandy
>>
>>
>> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi Sandy,
>>>
>>> Yes, I have been using AMRMClient APIs. I am planning to shift to
>>> whatever way is this white list feature is supported with. But am not sure
>>> what is meant by submitting ResourceRequests directly to RM. Can you please
>>> elaborate on this or give me a pointer to some example code on how to do
>>> it...
>>>
>>> Thanks for the reply,
>>>
>>> -Kishore
>>>
>>>
>>> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com>wrote:
>>>
>>>> Hi Krishna,
>>>>
>>>> From your previous email, it looks like you are using the AMRMClient
>>>> APIs. Support for whitelisting is not yet supported through them. I am
>>>> working on this in YARN-521, which should be included in the next release
>>>> after 2.1.0-beta. If you are submitting ResourceRequests directly to the
>>>> RM, you can whitelist a node by
>>>> * setting the relaxLocality flag on the node-level ResourceRequest to
>>>> true
>>>> * setting the relaxLocality flag on the corresponding rack-level
>>>> ResourceRequest to false
>>>> * setting the relaxLocality flag on the corresponding any-level
>>>> ResourceRequest to false
>>>>
>>>> -Sandy
>>>>
>>>>
>>>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>>>> write2kishore@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Can someone please point to some example code of how to use the
>>>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>>>> and want to use this feature.
>>>>>
>>>>> It would be great if you can point me to some description of what
>>>>> this white listing feature is, I have gone through some JIRA logs related
>>>>> to this but more concrete explanation would be helpful.
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>
>>>>
>>>
>>
>
Re: whitelist feature of YARN
Posted by Sandy Ryza <sa...@cloudera.com>.
Responses inline:
On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:
> Hi Sandy,
>
> Thanks for the reply and it is good to know YARN-521 is done! Please
> answer my following questions
>
> 1) when is 2.1.0-beta going to be released? is it soon or do you suggest
> me take it from the trunk or is there a recent release candidate available?
>
> We're very close and my guess would be no later than the end of the
month (don't hold me to this).
> 2) I have recently changed my application to use the new Asynchronous
> interfaces. I am hoping it works with that too, correct me if I am wrong.
>
ContainerRequest is shared by the async interfaces as well so it should
work here.
>
> 3) Change in interface:
>
> The old interface for ContainerRequest constructor used to be this:
>
> public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority, int containerCount);
>
> where as now it is changed to
>
> a) public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority)
> &
>
> b) public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority, boolean relaxLocality)
>
> that means the old argument containerCount is gone! How would I be able to
> specify how many containers do I need?
>
> We now expect that you submit a ContainerRequest for each container you
want.
> -Kishore
>
>
>
>
> On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> YARN-521, which brings whitelisting to the AMRMClient APIs, is now
>> included in 2.1.0-beta. Check out the doc for the relaxLocality paramater
>> in ContainerRequest in AMRMClient:
>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and
>> I can help clarify here if anything's confusing.
>>
>> -Sandy
>>
>>
>> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi Sandy,
>>>
>>> Yes, I have been using AMRMClient APIs. I am planning to shift to
>>> whatever way is this white list feature is supported with. But am not sure
>>> what is meant by submitting ResourceRequests directly to RM. Can you please
>>> elaborate on this or give me a pointer to some example code on how to do
>>> it...
>>>
>>> Thanks for the reply,
>>>
>>> -Kishore
>>>
>>>
>>> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com>wrote:
>>>
>>>> Hi Krishna,
>>>>
>>>> From your previous email, it looks like you are using the AMRMClient
>>>> APIs. Support for whitelisting is not yet supported through them. I am
>>>> working on this in YARN-521, which should be included in the next release
>>>> after 2.1.0-beta. If you are submitting ResourceRequests directly to the
>>>> RM, you can whitelist a node by
>>>> * setting the relaxLocality flag on the node-level ResourceRequest to
>>>> true
>>>> * setting the relaxLocality flag on the corresponding rack-level
>>>> ResourceRequest to false
>>>> * setting the relaxLocality flag on the corresponding any-level
>>>> ResourceRequest to false
>>>>
>>>> -Sandy
>>>>
>>>>
>>>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>>>> write2kishore@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Can someone please point to some example code of how to use the
>>>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>>>> and want to use this feature.
>>>>>
>>>>> It would be great if you can point me to some description of what
>>>>> this white listing feature is, I have gone through some JIRA logs related
>>>>> to this but more concrete explanation would be helpful.
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>
>>>>
>>>
>>
>
Re: whitelist feature of YARN
Posted by Sandy Ryza <sa...@cloudera.com>.
Responses inline:
On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:
> Hi Sandy,
>
> Thanks for the reply and it is good to know YARN-521 is done! Please
> answer my following questions
>
> 1) when is 2.1.0-beta going to be released? is it soon or do you suggest
> me take it from the trunk or is there a recent release candidate available?
>
> We're very close and my guess would be no later than the end of the
month (don't hold me to this).
> 2) I have recently changed my application to use the new Asynchronous
> interfaces. I am hoping it works with that too, correct me if I am wrong.
>
ContainerRequest is shared by the async interfaces as well so it should
work here.
>
> 3) Change in interface:
>
> The old interface for ContainerRequest constructor used to be this:
>
> public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority, int containerCount);
>
> where as now it is changed to
>
> a) public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority)
> &
>
> b) public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority, boolean relaxLocality)
>
> that means the old argument containerCount is gone! How would I be able to
> specify how many containers do I need?
>
> We now expect that you submit a ContainerRequest for each container you
want.
> -Kishore
>
>
>
>
> On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> YARN-521, which brings whitelisting to the AMRMClient APIs, is now
>> included in 2.1.0-beta. Check out the doc for the relaxLocality paramater
>> in ContainerRequest in AMRMClient:
>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and
>> I can help clarify here if anything's confusing.
>>
>> -Sandy
>>
>>
>> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi Sandy,
>>>
>>> Yes, I have been using AMRMClient APIs. I am planning to shift to
>>> whatever way is this white list feature is supported with. But am not sure
>>> what is meant by submitting ResourceRequests directly to RM. Can you please
>>> elaborate on this or give me a pointer to some example code on how to do
>>> it...
>>>
>>> Thanks for the reply,
>>>
>>> -Kishore
>>>
>>>
>>> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com>wrote:
>>>
>>>> Hi Krishna,
>>>>
>>>> From your previous email, it looks like you are using the AMRMClient
>>>> APIs. Support for whitelisting is not yet supported through them. I am
>>>> working on this in YARN-521, which should be included in the next release
>>>> after 2.1.0-beta. If you are submitting ResourceRequests directly to the
>>>> RM, you can whitelist a node by
>>>> * setting the relaxLocality flag on the node-level ResourceRequest to
>>>> true
>>>> * setting the relaxLocality flag on the corresponding rack-level
>>>> ResourceRequest to false
>>>> * setting the relaxLocality flag on the corresponding any-level
>>>> ResourceRequest to false
>>>>
>>>> -Sandy
>>>>
>>>>
>>>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>>>> write2kishore@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Can someone please point to some example code of how to use the
>>>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>>>> and want to use this feature.
>>>>>
>>>>> It would be great if you can point me to some description of what
>>>>> this white listing feature is, I have gone through some JIRA logs related
>>>>> to this but more concrete explanation would be helpful.
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>
>>>>
>>>
>>
>
Re: whitelist feature of YARN
Posted by Sandy Ryza <sa...@cloudera.com>.
Responses inline:
On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:
> Hi Sandy,
>
> Thanks for the reply and it is good to know YARN-521 is done! Please
> answer my following questions
>
> 1) when is 2.1.0-beta going to be released? is it soon or do you suggest
> me take it from the trunk or is there a recent release candidate available?
>
> We're very close and my guess would be no later than the end of the
month (don't hold me to this).
> 2) I have recently changed my application to use the new Asynchronous
> interfaces. I am hoping it works with that too, correct me if I am wrong.
>
ContainerRequest is shared by the async interfaces as well so it should
work here.
>
> 3) Change in interface:
>
> The old interface for ContainerRequest constructor used to be this:
>
> public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority, int containerCount);
>
> where as now it is changed to
>
> a) public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority)
> &
>
> b) public ContainerRequest(Resource capability, String[] nodes,
> String[] racks, Priority priority, boolean relaxLocality)
>
> that means the old argument containerCount is gone! How would I be able to
> specify how many containers do I need?
>
> We now expect that you submit a ContainerRequest for each container you
want.
> -Kishore
>
>
>
>
> On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> YARN-521, which brings whitelisting to the AMRMClient APIs, is now
>> included in 2.1.0-beta. Check out the doc for the relaxLocality paramater
>> in ContainerRequest in AMRMClient:
>> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and
>> I can help clarify here if anything's confusing.
>>
>> -Sandy
>>
>>
>> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi Sandy,
>>>
>>> Yes, I have been using AMRMClient APIs. I am planning to shift to
>>> whatever way is this white list feature is supported with. But am not sure
>>> what is meant by submitting ResourceRequests directly to RM. Can you please
>>> elaborate on this or give me a pointer to some example code on how to do
>>> it...
>>>
>>> Thanks for the reply,
>>>
>>> -Kishore
>>>
>>>
>>> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com>wrote:
>>>
>>>> Hi Krishna,
>>>>
>>>> From your previous email, it looks like you are using the AMRMClient
>>>> APIs. Support for whitelisting is not yet supported through them. I am
>>>> working on this in YARN-521, which should be included in the next release
>>>> after 2.1.0-beta. If you are submitting ResourceRequests directly to the
>>>> RM, you can whitelist a node by
>>>> * setting the relaxLocality flag on the node-level ResourceRequest to
>>>> true
>>>> * setting the relaxLocality flag on the corresponding rack-level
>>>> ResourceRequest to false
>>>> * setting the relaxLocality flag on the corresponding any-level
>>>> ResourceRequest to false
>>>>
>>>> -Sandy
>>>>
>>>>
>>>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>>>> write2kishore@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Can someone please point to some example code of how to use the
>>>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>>>> and want to use this feature.
>>>>>
>>>>> It would be great if you can point me to some description of what
>>>>> this white listing feature is, I have gone through some JIRA logs related
>>>>> to this but more concrete explanation would be helpful.
>>>>>
>>>>> Thanks,
>>>>> Kishore
>>>>>
>>>>
>>>>
>>>
>>
>
Re: whitelist feature of YARN
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Hi Sandy,
Thanks for the reply and it is good to know YARN-521 is done! Please
answer my following questions
1) when is 2.1.0-beta going to be released? is it soon or do you suggest me
take it from the trunk or is there a recent release candidate available?
2) I have recently changed my application to use the new Asynchronous
interfaces. I am hoping it works with that too, correct me if I am wrong.
3) Change in interface:
The old interface for ContainerRequest constructor used to be this:
public ContainerRequest(Resource capability, String[] nodes,
String[] racks, Priority priority, int containerCount);
where as now it is changed to
a) public ContainerRequest(Resource capability, String[] nodes,
String[] racks, Priority priority)
&
b) public ContainerRequest(Resource capability, String[] nodes,
String[] racks, Priority priority, boolean relaxLocality)
that means the old argument containerCount is gone! How would I be able to
specify how many containers do I need?
-Kishore
On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> YARN-521, which brings whitelisting to the AMRMClient APIs, is now
> included in 2.1.0-beta. Check out the doc for the relaxLocality paramater
> in ContainerRequest in AMRMClient:
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and
> I can help clarify here if anything's confusing.
>
> -Sandy
>
>
> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Sandy,
>>
>> Yes, I have been using AMRMClient APIs. I am planning to shift to
>> whatever way is this white list feature is supported with. But am not sure
>> what is meant by submitting ResourceRequests directly to RM. Can you please
>> elaborate on this or give me a pointer to some example code on how to do
>> it...
>>
>> Thanks for the reply,
>>
>> -Kishore
>>
>>
>> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com>wrote:
>>
>>> Hi Krishna,
>>>
>>> From your previous email, it looks like you are using the AMRMClient
>>> APIs. Support for whitelisting is not yet supported through them. I am
>>> working on this in YARN-521, which should be included in the next release
>>> after 2.1.0-beta. If you are submitting ResourceRequests directly to the
>>> RM, you can whitelist a node by
>>> * setting the relaxLocality flag on the node-level ResourceRequest to
>>> true
>>> * setting the relaxLocality flag on the corresponding rack-level
>>> ResourceRequest to false
>>> * setting the relaxLocality flag on the corresponding any-level
>>> ResourceRequest to false
>>>
>>> -Sandy
>>>
>>>
>>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Can someone please point to some example code of how to use the
>>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>>> and want to use this feature.
>>>>
>>>> It would be great if you can point me to some description of what
>>>> this white listing feature is, I have gone through some JIRA logs related
>>>> to this but more concrete explanation would be helpful.
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>
>>>
>>
>
Re: whitelist feature of YARN
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Hi Sandy,
Thanks for the reply and it is good to know YARN-521 is done! Please
answer my following questions
1) when is 2.1.0-beta going to be released? is it soon or do you suggest me
take it from the trunk or is there a recent release candidate available?
2) I have recently changed my application to use the new Asynchronous
interfaces. I am hoping it works with that too, correct me if I am wrong.
3) Change in interface:
The old interface for ContainerRequest constructor used to be this:
public ContainerRequest(Resource capability, String[] nodes,
String[] racks, Priority priority, int containerCount);
where as now it is changed to
a) public ContainerRequest(Resource capability, String[] nodes,
String[] racks, Priority priority)
&
b) public ContainerRequest(Resource capability, String[] nodes,
String[] racks, Priority priority, boolean relaxLocality)
that means the old argument containerCount is gone! How would I be able to
specify how many containers do I need?
-Kishore
On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> YARN-521, which brings whitelisting to the AMRMClient APIs, is now
> included in 2.1.0-beta. Check out the doc for the relaxLocality paramater
> in ContainerRequest in AMRMClient:
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and
> I can help clarify here if anything's confusing.
>
> -Sandy
>
>
> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Sandy,
>>
>> Yes, I have been using AMRMClient APIs. I am planning to shift to
>> whatever way is this white list feature is supported with. But am not sure
>> what is meant by submitting ResourceRequests directly to RM. Can you please
>> elaborate on this or give me a pointer to some example code on how to do
>> it...
>>
>> Thanks for the reply,
>>
>> -Kishore
>>
>>
>> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com>wrote:
>>
>>> Hi Krishna,
>>>
>>> From your previous email, it looks like you are using the AMRMClient
>>> APIs. Support for whitelisting is not yet supported through them. I am
>>> working on this in YARN-521, which should be included in the next release
>>> after 2.1.0-beta. If you are submitting ResourceRequests directly to the
>>> RM, you can whitelist a node by
>>> * setting the relaxLocality flag on the node-level ResourceRequest to
>>> true
>>> * setting the relaxLocality flag on the corresponding rack-level
>>> ResourceRequest to false
>>> * setting the relaxLocality flag on the corresponding any-level
>>> ResourceRequest to false
>>>
>>> -Sandy
>>>
>>>
>>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Can someone please point to some example code of how to use the
>>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>>> and want to use this feature.
>>>>
>>>> It would be great if you can point me to some description of what
>>>> this white listing feature is, I have gone through some JIRA logs related
>>>> to this but more concrete explanation would be helpful.
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>
>>>
>>
>
Re: whitelist feature of YARN
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Hi Sandy,
Thanks for the reply and it is good to know YARN-521 is done! Please
answer my following questions
1) when is 2.1.0-beta going to be released? is it soon or do you suggest me
take it from the trunk or is there a recent release candidate available?
2) I have recently changed my application to use the new Asynchronous
interfaces. I am hoping it works with that too, correct me if I am wrong.
3) Change in interface:
The old interface for ContainerRequest constructor used to be this:
public ContainerRequest(Resource capability, String[] nodes,
String[] racks, Priority priority, int containerCount);
where as now it is changed to
a) public ContainerRequest(Resource capability, String[] nodes,
String[] racks, Priority priority)
&
b) public ContainerRequest(Resource capability, String[] nodes,
String[] racks, Priority priority, boolean relaxLocality)
that means the old argument containerCount is gone! How would I be able to
specify how many containers do I need?
-Kishore
On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> YARN-521, which brings whitelisting to the AMRMClient APIs, is now
> included in 2.1.0-beta. Check out the doc for the relaxLocality paramater
> in ContainerRequest in AMRMClient:
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and
> I can help clarify here if anything's confusing.
>
> -Sandy
>
>
> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Sandy,
>>
>> Yes, I have been using AMRMClient APIs. I am planning to shift to
>> whatever way is this white list feature is supported with. But am not sure
>> what is meant by submitting ResourceRequests directly to RM. Can you please
>> elaborate on this or give me a pointer to some example code on how to do
>> it...
>>
>> Thanks for the reply,
>>
>> -Kishore
>>
>>
>> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com>wrote:
>>
>>> Hi Krishna,
>>>
>>> From your previous email, it looks like you are using the AMRMClient
>>> APIs. Support for whitelisting is not yet supported through them. I am
>>> working on this in YARN-521, which should be included in the next release
>>> after 2.1.0-beta. If you are submitting ResourceRequests directly to the
>>> RM, you can whitelist a node by
>>> * setting the relaxLocality flag on the node-level ResourceRequest to
>>> true
>>> * setting the relaxLocality flag on the corresponding rack-level
>>> ResourceRequest to false
>>> * setting the relaxLocality flag on the corresponding any-level
>>> ResourceRequest to false
>>>
>>> -Sandy
>>>
>>>
>>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Can someone please point to some example code of how to use the
>>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>>> and want to use this feature.
>>>>
>>>> It would be great if you can point me to some description of what
>>>> this white listing feature is, I have gone through some JIRA logs related
>>>> to this but more concrete explanation would be helpful.
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>
>>>
>>
>
Re: whitelist feature of YARN
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Hi Sandy,
Thanks for the reply and it is good to know YARN-521 is done! Please
answer my following questions
1) when is 2.1.0-beta going to be released? is it soon or do you suggest me
take it from the trunk or is there a recent release candidate available?
2) I have recently changed my application to use the new Asynchronous
interfaces. I am hoping it works with that too, correct me if I am wrong.
3) Change in interface:
The old interface for ContainerRequest constructor used to be this:
public ContainerRequest(Resource capability, String[] nodes,
String[] racks, Priority priority, int containerCount);
where as now it is changed to
a) public ContainerRequest(Resource capability, String[] nodes,
String[] racks, Priority priority)
&
b) public ContainerRequest(Resource capability, String[] nodes,
String[] racks, Priority priority, boolean relaxLocality)
that means the old argument containerCount is gone! How would I be able to
specify how many containers do I need?
-Kishore
On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> YARN-521, which brings whitelisting to the AMRMClient APIs, is now
> included in 2.1.0-beta. Check out the doc for the relaxLocality paramater
> in ContainerRequest in AMRMClient:
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java and
> I can help clarify here if anything's confusing.
>
> -Sandy
>
>
> On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi Sandy,
>>
>> Yes, I have been using AMRMClient APIs. I am planning to shift to
>> whatever way is this white list feature is supported with. But am not sure
>> what is meant by submitting ResourceRequests directly to RM. Can you please
>> elaborate on this or give me a pointer to some example code on how to do
>> it...
>>
>> Thanks for the reply,
>>
>> -Kishore
>>
>>
>> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com>wrote:
>>
>>> Hi Krishna,
>>>
>>> From your previous email, it looks like you are using the AMRMClient
>>> APIs. Support for whitelisting is not yet supported through them. I am
>>> working on this in YARN-521, which should be included in the next release
>>> after 2.1.0-beta. If you are submitting ResourceRequests directly to the
>>> RM, you can whitelist a node by
>>> * setting the relaxLocality flag on the node-level ResourceRequest to
>>> true
>>> * setting the relaxLocality flag on the corresponding rack-level
>>> ResourceRequest to false
>>> * setting the relaxLocality flag on the corresponding any-level
>>> ResourceRequest to false
>>>
>>> -Sandy
>>>
>>>
>>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>>> write2kishore@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Can someone please point to some example code of how to use the
>>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>>> and want to use this feature.
>>>>
>>>> It would be great if you can point me to some description of what
>>>> this white listing feature is, I have gone through some JIRA logs related
>>>> to this but more concrete explanation would be helpful.
>>>>
>>>> Thanks,
>>>> Kishore
>>>>
>>>
>>>
>>
>
Re: whitelist feature of YARN
Posted by Sandy Ryza <sa...@cloudera.com>.
YARN-521, which brings whitelisting to the AMRMClient APIs, is now included
in 2.1.0-beta. Check out the doc for the relaxLocality paramater in
ContainerRequest in AMRMClient:
https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java
and
I can help clarify here if anything's confusing.
-Sandy
On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:
> Hi Sandy,
>
> Yes, I have been using AMRMClient APIs. I am planning to shift to
> whatever way is this white list feature is supported with. But am not sure
> what is meant by submitting ResourceRequests directly to RM. Can you please
> elaborate on this or give me a pointer to some example code on how to do
> it...
>
> Thanks for the reply,
>
> -Kishore
>
>
> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Hi Krishna,
>>
>> From your previous email, it looks like you are using the AMRMClient
>> APIs. Support for whitelisting is not yet supported through them. I am
>> working on this in YARN-521, which should be included in the next release
>> after 2.1.0-beta. If you are submitting ResourceRequests directly to the
>> RM, you can whitelist a node by
>> * setting the relaxLocality flag on the node-level ResourceRequest to true
>> * setting the relaxLocality flag on the corresponding rack-level
>> ResourceRequest to false
>> * setting the relaxLocality flag on the corresponding any-level
>> ResourceRequest to false
>>
>> -Sandy
>>
>>
>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Can someone please point to some example code of how to use the
>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>> and want to use this feature.
>>>
>>> It would be great if you can point me to some description of what this
>>> white listing feature is, I have gone through some JIRA logs related to
>>> this but more concrete explanation would be helpful.
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>
Re: whitelist feature of YARN
Posted by Sandy Ryza <sa...@cloudera.com>.
YARN-521, which brings whitelisting to the AMRMClient APIs, is now included
in 2.1.0-beta. Check out the doc for the relaxLocality paramater in
ContainerRequest in AMRMClient:
https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java
and
I can help clarify here if anything's confusing.
-Sandy
On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:
> Hi Sandy,
>
> Yes, I have been using AMRMClient APIs. I am planning to shift to
> whatever way is this white list feature is supported with. But am not sure
> what is meant by submitting ResourceRequests directly to RM. Can you please
> elaborate on this or give me a pointer to some example code on how to do
> it...
>
> Thanks for the reply,
>
> -Kishore
>
>
> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Hi Krishna,
>>
>> From your previous email, it looks like you are using the AMRMClient
>> APIs. Support for whitelisting is not yet supported through them. I am
>> working on this in YARN-521, which should be included in the next release
>> after 2.1.0-beta. If you are submitting ResourceRequests directly to the
>> RM, you can whitelist a node by
>> * setting the relaxLocality flag on the node-level ResourceRequest to true
>> * setting the relaxLocality flag on the corresponding rack-level
>> ResourceRequest to false
>> * setting the relaxLocality flag on the corresponding any-level
>> ResourceRequest to false
>>
>> -Sandy
>>
>>
>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Can someone please point to some example code of how to use the
>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>> and want to use this feature.
>>>
>>> It would be great if you can point me to some description of what this
>>> white listing feature is, I have gone through some JIRA logs related to
>>> this but more concrete explanation would be helpful.
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>
Re: whitelist feature of YARN
Posted by Sandy Ryza <sa...@cloudera.com>.
YARN-521, which brings whitelisting to the AMRMClient APIs, is now included
in 2.1.0-beta. Check out the doc for the relaxLocality paramater in
ContainerRequest in AMRMClient:
https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java
and
I can help clarify here if anything's confusing.
-Sandy
On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:
> Hi Sandy,
>
> Yes, I have been using AMRMClient APIs. I am planning to shift to
> whatever way is this white list feature is supported with. But am not sure
> what is meant by submitting ResourceRequests directly to RM. Can you please
> elaborate on this or give me a pointer to some example code on how to do
> it...
>
> Thanks for the reply,
>
> -Kishore
>
>
> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Hi Krishna,
>>
>> From your previous email, it looks like you are using the AMRMClient
>> APIs. Support for whitelisting is not yet supported through them. I am
>> working on this in YARN-521, which should be included in the next release
>> after 2.1.0-beta. If you are submitting ResourceRequests directly to the
>> RM, you can whitelist a node by
>> * setting the relaxLocality flag on the node-level ResourceRequest to true
>> * setting the relaxLocality flag on the corresponding rack-level
>> ResourceRequest to false
>> * setting the relaxLocality flag on the corresponding any-level
>> ResourceRequest to false
>>
>> -Sandy
>>
>>
>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Can someone please point to some example code of how to use the
>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>> and want to use this feature.
>>>
>>> It would be great if you can point me to some description of what this
>>> white listing feature is, I have gone through some JIRA logs related to
>>> this but more concrete explanation would be helpful.
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>
Re: whitelist feature of YARN
Posted by Sandy Ryza <sa...@cloudera.com>.
YARN-521, which brings whitelisting to the AMRMClient APIs, is now included
in 2.1.0-beta. Check out the doc for the relaxLocality paramater in
ContainerRequest in AMRMClient:
https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java
and
I can help clarify here if anything's confusing.
-Sandy
On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:
> Hi Sandy,
>
> Yes, I have been using AMRMClient APIs. I am planning to shift to
> whatever way is this white list feature is supported with. But am not sure
> what is meant by submitting ResourceRequests directly to RM. Can you please
> elaborate on this or give me a pointer to some example code on how to do
> it...
>
> Thanks for the reply,
>
> -Kishore
>
>
> On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Hi Krishna,
>>
>> From your previous email, it looks like you are using the AMRMClient
>> APIs. Support for whitelisting is not yet supported through them. I am
>> working on this in YARN-521, which should be included in the next release
>> after 2.1.0-beta. If you are submitting ResourceRequests directly to the
>> RM, you can whitelist a node by
>> * setting the relaxLocality flag on the node-level ResourceRequest to true
>> * setting the relaxLocality flag on the corresponding rack-level
>> ResourceRequest to false
>> * setting the relaxLocality flag on the corresponding any-level
>> ResourceRequest to false
>>
>> -Sandy
>>
>>
>> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Can someone please point to some example code of how to use the
>>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>>> and want to use this feature.
>>>
>>> It would be great if you can point me to some description of what this
>>> white listing feature is, I have gone through some JIRA logs related to
>>> this but more concrete explanation would be helpful.
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>
Re: whitelist feature of YARN
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Hi Sandy,
Yes, I have been using AMRMClient APIs. I am planning to shift to
whatever way is this white list feature is supported with. But am not sure
what is meant by submitting ResourceRequests directly to RM. Can you please
elaborate on this or give me a pointer to some example code on how to do
it...
Thanks for the reply,
-Kishore
On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Krishna,
>
> From your previous email, it looks like you are using the AMRMClient APIs.
> Support for whitelisting is not yet supported through them. I am working
> on this in YARN-521, which should be included in the next release after
> 2.1.0-beta. If you are submitting ResourceRequests directly to the RM, you
> can whitelist a node by
> * setting the relaxLocality flag on the node-level ResourceRequest to true
> * setting the relaxLocality flag on the corresponding rack-level
> ResourceRequest to false
> * setting the relaxLocality flag on the corresponding any-level
> ResourceRequest to false
>
> -Sandy
>
>
> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>>
>> Can someone please point to some example code of how to use the
>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>> and want to use this feature.
>>
>> It would be great if you can point me to some description of what this
>> white listing feature is, I have gone through some JIRA logs related to
>> this but more concrete explanation would be helpful.
>>
>> Thanks,
>> Kishore
>>
>
>
Re: whitelist feature of YARN
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Hi Sandy,
Yes, I have been using AMRMClient APIs. I am planning to shift to
whatever way is this white list feature is supported with. But am not sure
what is meant by submitting ResourceRequests directly to RM. Can you please
elaborate on this or give me a pointer to some example code on how to do
it...
Thanks for the reply,
-Kishore
On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Krishna,
>
> From your previous email, it looks like you are using the AMRMClient APIs.
> Support for whitelisting is not yet supported through them. I am working
> on this in YARN-521, which should be included in the next release after
> 2.1.0-beta. If you are submitting ResourceRequests directly to the RM, you
> can whitelist a node by
> * setting the relaxLocality flag on the node-level ResourceRequest to true
> * setting the relaxLocality flag on the corresponding rack-level
> ResourceRequest to false
> * setting the relaxLocality flag on the corresponding any-level
> ResourceRequest to false
>
> -Sandy
>
>
> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>>
>> Can someone please point to some example code of how to use the
>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>> and want to use this feature.
>>
>> It would be great if you can point me to some description of what this
>> white listing feature is, I have gone through some JIRA logs related to
>> this but more concrete explanation would be helpful.
>>
>> Thanks,
>> Kishore
>>
>
>
Re: whitelist feature of YARN
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Hi Sandy,
Yes, I have been using AMRMClient APIs. I am planning to shift to
whatever way is this white list feature is supported with. But am not sure
what is meant by submitting ResourceRequests directly to RM. Can you please
elaborate on this or give me a pointer to some example code on how to do
it...
Thanks for the reply,
-Kishore
On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Krishna,
>
> From your previous email, it looks like you are using the AMRMClient APIs.
> Support for whitelisting is not yet supported through them. I am working
> on this in YARN-521, which should be included in the next release after
> 2.1.0-beta. If you are submitting ResourceRequests directly to the RM, you
> can whitelist a node by
> * setting the relaxLocality flag on the node-level ResourceRequest to true
> * setting the relaxLocality flag on the corresponding rack-level
> ResourceRequest to false
> * setting the relaxLocality flag on the corresponding any-level
> ResourceRequest to false
>
> -Sandy
>
>
> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>>
>> Can someone please point to some example code of how to use the
>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>> and want to use this feature.
>>
>> It would be great if you can point me to some description of what this
>> white listing feature is, I have gone through some JIRA logs related to
>> this but more concrete explanation would be helpful.
>>
>> Thanks,
>> Kishore
>>
>
>
Re: whitelist feature of YARN
Posted by Krishna Kishore Bonagiri <wr...@gmail.com>.
Hi Sandy,
Yes, I have been using AMRMClient APIs. I am planning to shift to
whatever way is this white list feature is supported with. But am not sure
what is meant by submitting ResourceRequests directly to RM. Can you please
elaborate on this or give me a pointer to some example code on how to do
it...
Thanks for the reply,
-Kishore
On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Krishna,
>
> From your previous email, it looks like you are using the AMRMClient APIs.
> Support for whitelisting is not yet supported through them. I am working
> on this in YARN-521, which should be included in the next release after
> 2.1.0-beta. If you are submitting ResourceRequests directly to the RM, you
> can whitelist a node by
> * setting the relaxLocality flag on the node-level ResourceRequest to true
> * setting the relaxLocality flag on the corresponding rack-level
> ResourceRequest to false
> * setting the relaxLocality flag on the corresponding any-level
> ResourceRequest to false
>
> -Sandy
>
>
> On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>
>> Hi,
>>
>> Can someone please point to some example code of how to use the
>> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
>> and want to use this feature.
>>
>> It would be great if you can point me to some description of what this
>> white listing feature is, I have gone through some JIRA logs related to
>> this but more concrete explanation would be helpful.
>>
>> Thanks,
>> Kishore
>>
>
>
Re: whitelist feature of YARN
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Krishna,
>From your previous email, it looks like you are using the AMRMClient APIs.
Support for whitelisting is not yet supported through them. I am working
on this in YARN-521, which should be included in the next release after
2.1.0-beta. If you are submitting ResourceRequests directly to the RM, you
can whitelist a node by
* setting the relaxLocality flag on the node-level ResourceRequest to true
* setting the relaxLocality flag on the corresponding rack-level
ResourceRequest to false
* setting the relaxLocality flag on the corresponding any-level
ResourceRequest to false
-Sandy
On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:
> Hi,
>
> Can someone please point to some example code of how to use the
> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
> and want to use this feature.
>
> It would be great if you can point me to some description of what this
> white listing feature is, I have gone through some JIRA logs related to
> this but more concrete explanation would be helpful.
>
> Thanks,
> Kishore
>
Re: whitelist feature of YARN
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Krishna,
>From your previous email, it looks like you are using the AMRMClient APIs.
Support for whitelisting is not yet supported through them. I am working
on this in YARN-521, which should be included in the next release after
2.1.0-beta. If you are submitting ResourceRequests directly to the RM, you
can whitelist a node by
* setting the relaxLocality flag on the node-level ResourceRequest to true
* setting the relaxLocality flag on the corresponding rack-level
ResourceRequest to false
* setting the relaxLocality flag on the corresponding any-level
ResourceRequest to false
-Sandy
On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:
> Hi,
>
> Can someone please point to some example code of how to use the
> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
> and want to use this feature.
>
> It would be great if you can point me to some description of what this
> white listing feature is, I have gone through some JIRA logs related to
> this but more concrete explanation would be helpful.
>
> Thanks,
> Kishore
>
Re: whitelist feature of YARN
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Krishna,
>From your previous email, it looks like you are using the AMRMClient APIs.
Support for whitelisting is not yet supported through them. I am working
on this in YARN-521, which should be included in the next release after
2.1.0-beta. If you are submitting ResourceRequests directly to the RM, you
can whitelist a node by
* setting the relaxLocality flag on the node-level ResourceRequest to true
* setting the relaxLocality flag on the corresponding rack-level
ResourceRequest to false
* setting the relaxLocality flag on the corresponding any-level
ResourceRequest to false
-Sandy
On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:
> Hi,
>
> Can someone please point to some example code of how to use the
> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
> and want to use this feature.
>
> It would be great if you can point me to some description of what this
> white listing feature is, I have gone through some JIRA logs related to
> this but more concrete explanation would be helpful.
>
> Thanks,
> Kishore
>
Re: whitelist feature of YARN
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Krishna,
>From your previous email, it looks like you are using the AMRMClient APIs.
Support for whitelisting is not yet supported through them. I am working
on this in YARN-521, which should be included in the next release after
2.1.0-beta. If you are submitting ResourceRequests directly to the RM, you
can whitelist a node by
* setting the relaxLocality flag on the node-level ResourceRequest to true
* setting the relaxLocality flag on the corresponding rack-level
ResourceRequest to false
* setting the relaxLocality flag on the corresponding any-level
ResourceRequest to false
-Sandy
On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <
write2kishore@gmail.com> wrote:
> Hi,
>
> Can someone please point to some example code of how to use the
> whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta
> and want to use this feature.
>
> It would be great if you can point me to some description of what this
> white listing feature is, I have gone through some JIRA logs related to
> this but more concrete explanation would be helpful.
>
> Thanks,
> Kishore
>