You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Brian C. Huffman" <bh...@etinternational.com> on 2013/04/11 20:00:43 UTC
YARN - specify hosts in ContainerRequest
All,
I'm starting to work with YARN and I'm trying to make some changes to
the distributedshell example.
Specifically I'm trying to set the host where the distributed shell
container will run. I added the following code into
ApplicationMaster.java:
final String[] containerNodes = { "node1" };
LOG.info("Trying to request node: " + containerNodes[0]);
ContainerRequest request = new ContainerRequest(capability,
containerNodes, null,
pri, numContainers);
And while the log output shows this code is being executed ("Trying to
request..."), it doesn't seem to use it. Is this functionality that
isn't complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
Thanks,
Brian
Re: YARN - specify hosts in ContainerRequest
Posted by Sandy Ryza <sa...@cloudera.com>.
YARN currently does not provide any guarantees that a container will be
placed where you ask for it. This is being addressed in YARN-392 and
YARN-398. For now, the Fair Scheduler and Capacity Scheduler will make a
best effort to do this through delay scheduling. If you use the Fair
Scheduler, you can turn on delay scheduling by setting
yarn.scheduler.fair.locality.threshold.node to a float between 0 and 1,
which corresponds to the number of node-RM heartbeats to delay allocating a
non-local container, as a fraction of the cluster size.
Sandy
On Fri, Apr 12, 2013 at 12:08 PM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> I get a container, but not on the node I'm asking for.
>
> Thanks,
> Brian
>
>
> On 04/12/2013 03:01 PM, Sandy Ryza wrote:
>
> What do you mean when you say it doesn't seem to use the code? That
> you're not getting containers back?
>
> -Sandy
>
> On Fri, Apr 12, 2013 at 7:44 AM, Brian C. Huffman <
> bhuffman@etinternational.com> wrote:
>
>> Yes, this is exactly the DistributedShell example in the 2.0.3 source
>> (except for the added specification of containerNode).
>>
>> So, it's doing the following:
>> ContainerRequest containerAsk =
>> setupContainerAskForRM(askCount);
>> resourceManager.addContainerRequest(containerAsk);
>>
>>
>> // Send the request to RM
>> LOG.info("Asking RM for containers" + ", askCount=" + askCount);
>> AMResponse amResp = sendContainerAskToRM();
>>
>> FYI - the sendContainerAskToRM calls the allocate(progress).
>>
>> I don't see any NPEs in the resourcemanager's logs.
>>
>> Regards,
>> Brian
>>
>>
>> On 04/11/2013 02:50 PM, Harsh J wrote:
>>
>>> Hi Brian,
>>>
>>> Your way of creation seems fine. Are you "adding" this to the
>>> AMRMClient somewhere though, via [1], right before you call
>>> allocate(progress)? Cause otherwise this object's not used at all.
>>>
>>> P.s. I'm not sure we'll handle that null for racks all that well in
>>> certain places yet; I've noticed errors in my app when I've missed
>>> passing elements or passed them as nulls, but I didn't rebase it onto
>>> 2.0.3 yet so haven't reported anything so far - will do so soon. I'd
>>> check the RM log to see if there's any WARN logged there with an NPE.
>>>
>>> [1] -
>>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
>>>
>>> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
>>> <bh...@etinternational.com> wrote:
>>>
>>>> All,
>>>>
>>>> I'm starting to work with YARN and I'm trying to make some changes to
>>>> the
>>>> distributedshell example.
>>>>
>>>> Specifically I'm trying to set the host where the distributed shell
>>>> container will run. I added the following code into
>>>> ApplicationMaster.java:
>>>> final String[] containerNodes = { "node1" };
>>>>
>>>> LOG.info("Trying to request node: " + containerNodes[0]);
>>>> ContainerRequest request = new ContainerRequest(capability,
>>>> containerNodes, null,
>>>> pri, numContainers);
>>>>
>>>> And while the log output shows this code is being executed ("Trying to
>>>> request..."), it doesn't seem to use it. Is this functionality that
>>>> isn't
>>>> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>>>>
>>>> Thanks,
>>>> Brian
>>>>
>>>>
>>>
>>>
>>
>>
>
>
Re: YARN - specify hosts in ContainerRequest
Posted by Sandy Ryza <sa...@cloudera.com>.
YARN currently does not provide any guarantees that a container will be
placed where you ask for it. This is being addressed in YARN-392 and
YARN-398. For now, the Fair Scheduler and Capacity Scheduler will make a
best effort to do this through delay scheduling. If you use the Fair
Scheduler, you can turn on delay scheduling by setting
yarn.scheduler.fair.locality.threshold.node to a float between 0 and 1,
which corresponds to the number of node-RM heartbeats to delay allocating a
non-local container, as a fraction of the cluster size.
Sandy
On Fri, Apr 12, 2013 at 12:08 PM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> I get a container, but not on the node I'm asking for.
>
> Thanks,
> Brian
>
>
> On 04/12/2013 03:01 PM, Sandy Ryza wrote:
>
> What do you mean when you say it doesn't seem to use the code? That
> you're not getting containers back?
>
> -Sandy
>
> On Fri, Apr 12, 2013 at 7:44 AM, Brian C. Huffman <
> bhuffman@etinternational.com> wrote:
>
>> Yes, this is exactly the DistributedShell example in the 2.0.3 source
>> (except for the added specification of containerNode).
>>
>> So, it's doing the following:
>> ContainerRequest containerAsk =
>> setupContainerAskForRM(askCount);
>> resourceManager.addContainerRequest(containerAsk);
>>
>>
>> // Send the request to RM
>> LOG.info("Asking RM for containers" + ", askCount=" + askCount);
>> AMResponse amResp = sendContainerAskToRM();
>>
>> FYI - the sendContainerAskToRM calls the allocate(progress).
>>
>> I don't see any NPEs in the resourcemanager's logs.
>>
>> Regards,
>> Brian
>>
>>
>> On 04/11/2013 02:50 PM, Harsh J wrote:
>>
>>> Hi Brian,
>>>
>>> Your way of creation seems fine. Are you "adding" this to the
>>> AMRMClient somewhere though, via [1], right before you call
>>> allocate(progress)? Cause otherwise this object's not used at all.
>>>
>>> P.s. I'm not sure we'll handle that null for racks all that well in
>>> certain places yet; I've noticed errors in my app when I've missed
>>> passing elements or passed them as nulls, but I didn't rebase it onto
>>> 2.0.3 yet so haven't reported anything so far - will do so soon. I'd
>>> check the RM log to see if there's any WARN logged there with an NPE.
>>>
>>> [1] -
>>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
>>>
>>> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
>>> <bh...@etinternational.com> wrote:
>>>
>>>> All,
>>>>
>>>> I'm starting to work with YARN and I'm trying to make some changes to
>>>> the
>>>> distributedshell example.
>>>>
>>>> Specifically I'm trying to set the host where the distributed shell
>>>> container will run. I added the following code into
>>>> ApplicationMaster.java:
>>>> final String[] containerNodes = { "node1" };
>>>>
>>>> LOG.info("Trying to request node: " + containerNodes[0]);
>>>> ContainerRequest request = new ContainerRequest(capability,
>>>> containerNodes, null,
>>>> pri, numContainers);
>>>>
>>>> And while the log output shows this code is being executed ("Trying to
>>>> request..."), it doesn't seem to use it. Is this functionality that
>>>> isn't
>>>> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>>>>
>>>> Thanks,
>>>> Brian
>>>>
>>>>
>>>
>>>
>>
>>
>
>
Re: YARN - specify hosts in ContainerRequest
Posted by Sandy Ryza <sa...@cloudera.com>.
YARN currently does not provide any guarantees that a container will be
placed where you ask for it. This is being addressed in YARN-392 and
YARN-398. For now, the Fair Scheduler and Capacity Scheduler will make a
best effort to do this through delay scheduling. If you use the Fair
Scheduler, you can turn on delay scheduling by setting
yarn.scheduler.fair.locality.threshold.node to a float between 0 and 1,
which corresponds to the number of node-RM heartbeats to delay allocating a
non-local container, as a fraction of the cluster size.
Sandy
On Fri, Apr 12, 2013 at 12:08 PM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> I get a container, but not on the node I'm asking for.
>
> Thanks,
> Brian
>
>
> On 04/12/2013 03:01 PM, Sandy Ryza wrote:
>
> What do you mean when you say it doesn't seem to use the code? That
> you're not getting containers back?
>
> -Sandy
>
> On Fri, Apr 12, 2013 at 7:44 AM, Brian C. Huffman <
> bhuffman@etinternational.com> wrote:
>
>> Yes, this is exactly the DistributedShell example in the 2.0.3 source
>> (except for the added specification of containerNode).
>>
>> So, it's doing the following:
>> ContainerRequest containerAsk =
>> setupContainerAskForRM(askCount);
>> resourceManager.addContainerRequest(containerAsk);
>>
>>
>> // Send the request to RM
>> LOG.info("Asking RM for containers" + ", askCount=" + askCount);
>> AMResponse amResp = sendContainerAskToRM();
>>
>> FYI - the sendContainerAskToRM calls the allocate(progress).
>>
>> I don't see any NPEs in the resourcemanager's logs.
>>
>> Regards,
>> Brian
>>
>>
>> On 04/11/2013 02:50 PM, Harsh J wrote:
>>
>>> Hi Brian,
>>>
>>> Your way of creation seems fine. Are you "adding" this to the
>>> AMRMClient somewhere though, via [1], right before you call
>>> allocate(progress)? Cause otherwise this object's not used at all.
>>>
>>> P.s. I'm not sure we'll handle that null for racks all that well in
>>> certain places yet; I've noticed errors in my app when I've missed
>>> passing elements or passed them as nulls, but I didn't rebase it onto
>>> 2.0.3 yet so haven't reported anything so far - will do so soon. I'd
>>> check the RM log to see if there's any WARN logged there with an NPE.
>>>
>>> [1] -
>>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
>>>
>>> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
>>> <bh...@etinternational.com> wrote:
>>>
>>>> All,
>>>>
>>>> I'm starting to work with YARN and I'm trying to make some changes to
>>>> the
>>>> distributedshell example.
>>>>
>>>> Specifically I'm trying to set the host where the distributed shell
>>>> container will run. I added the following code into
>>>> ApplicationMaster.java:
>>>> final String[] containerNodes = { "node1" };
>>>>
>>>> LOG.info("Trying to request node: " + containerNodes[0]);
>>>> ContainerRequest request = new ContainerRequest(capability,
>>>> containerNodes, null,
>>>> pri, numContainers);
>>>>
>>>> And while the log output shows this code is being executed ("Trying to
>>>> request..."), it doesn't seem to use it. Is this functionality that
>>>> isn't
>>>> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>>>>
>>>> Thanks,
>>>> Brian
>>>>
>>>>
>>>
>>>
>>
>>
>
>
Re: YARN - specify hosts in ContainerRequest
Posted by Sandy Ryza <sa...@cloudera.com>.
YARN currently does not provide any guarantees that a container will be
placed where you ask for it. This is being addressed in YARN-392 and
YARN-398. For now, the Fair Scheduler and Capacity Scheduler will make a
best effort to do this through delay scheduling. If you use the Fair
Scheduler, you can turn on delay scheduling by setting
yarn.scheduler.fair.locality.threshold.node to a float between 0 and 1,
which corresponds to the number of node-RM heartbeats to delay allocating a
non-local container, as a fraction of the cluster size.
Sandy
On Fri, Apr 12, 2013 at 12:08 PM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> I get a container, but not on the node I'm asking for.
>
> Thanks,
> Brian
>
>
> On 04/12/2013 03:01 PM, Sandy Ryza wrote:
>
> What do you mean when you say it doesn't seem to use the code? That
> you're not getting containers back?
>
> -Sandy
>
> On Fri, Apr 12, 2013 at 7:44 AM, Brian C. Huffman <
> bhuffman@etinternational.com> wrote:
>
>> Yes, this is exactly the DistributedShell example in the 2.0.3 source
>> (except for the added specification of containerNode).
>>
>> So, it's doing the following:
>> ContainerRequest containerAsk =
>> setupContainerAskForRM(askCount);
>> resourceManager.addContainerRequest(containerAsk);
>>
>>
>> // Send the request to RM
>> LOG.info("Asking RM for containers" + ", askCount=" + askCount);
>> AMResponse amResp = sendContainerAskToRM();
>>
>> FYI - the sendContainerAskToRM calls the allocate(progress).
>>
>> I don't see any NPEs in the resourcemanager's logs.
>>
>> Regards,
>> Brian
>>
>>
>> On 04/11/2013 02:50 PM, Harsh J wrote:
>>
>>> Hi Brian,
>>>
>>> Your way of creation seems fine. Are you "adding" this to the
>>> AMRMClient somewhere though, via [1], right before you call
>>> allocate(progress)? Cause otherwise this object's not used at all.
>>>
>>> P.s. I'm not sure we'll handle that null for racks all that well in
>>> certain places yet; I've noticed errors in my app when I've missed
>>> passing elements or passed them as nulls, but I didn't rebase it onto
>>> 2.0.3 yet so haven't reported anything so far - will do so soon. I'd
>>> check the RM log to see if there's any WARN logged there with an NPE.
>>>
>>> [1] -
>>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
>>>
>>> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
>>> <bh...@etinternational.com> wrote:
>>>
>>>> All,
>>>>
>>>> I'm starting to work with YARN and I'm trying to make some changes to
>>>> the
>>>> distributedshell example.
>>>>
>>>> Specifically I'm trying to set the host where the distributed shell
>>>> container will run. I added the following code into
>>>> ApplicationMaster.java:
>>>> final String[] containerNodes = { "node1" };
>>>>
>>>> LOG.info("Trying to request node: " + containerNodes[0]);
>>>> ContainerRequest request = new ContainerRequest(capability,
>>>> containerNodes, null,
>>>> pri, numContainers);
>>>>
>>>> And while the log output shows this code is being executed ("Trying to
>>>> request..."), it doesn't seem to use it. Is this functionality that
>>>> isn't
>>>> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>>>>
>>>> Thanks,
>>>> Brian
>>>>
>>>>
>>>
>>>
>>
>>
>
>
Re: YARN - specify hosts in ContainerRequest
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
I get a container, but not on the node I'm asking for.
Thanks,
Brian
On 04/12/2013 03:01 PM, Sandy Ryza wrote:
> What do you mean when you say it doesn't seem to use the code? That
> you're not getting containers back?
>
> -Sandy
>
> On Fri, Apr 12, 2013 at 7:44 AM, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> Yes, this is exactly the DistributedShell example in the 2.0.3
> source (except for the added specification of containerNode).
>
> So, it's doing the following:
> ContainerRequest containerAsk =
> setupContainerAskForRM(askCount);
> resourceManager.addContainerRequest(containerAsk);
>
>
> // Send the request to RM
> LOG.info("Asking RM for containers" + ", askCount=" +
> askCount);
> AMResponse amResp = sendContainerAskToRM();
>
> FYI - the sendContainerAskToRM calls the allocate(progress).
>
> I don't see any NPEs in the resourcemanager's logs.
>
> Regards,
> Brian
>
>
> On 04/11/2013 02:50 PM, Harsh J wrote:
>
> Hi Brian,
>
> Your way of creation seems fine. Are you "adding" this to the
> AMRMClient somewhere though, via [1], right before you call
> allocate(progress)? Cause otherwise this object's not used at all.
>
> P.s. I'm not sure we'll handle that null for racks all that
> well in
> certain places yet; I've noticed errors in my app when I've missed
> passing elements or passed them as nulls, but I didn't rebase
> it onto
> 2.0.3 yet so haven't reported anything so far - will do so
> soon. I'd
> check the RM log to see if there's any WARN logged there with
> an NPE.
>
> [1] -
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
> <http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest%28org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest%29>
>
> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
> <bhuffman@etinternational.com
> <ma...@etinternational.com>> wrote:
>
> All,
>
> I'm starting to work with YARN and I'm trying to make some
> changes to the
> distributedshell example.
>
> Specifically I'm trying to set the host where the
> distributed shell
> container will run. I added the following code into
> ApplicationMaster.java:
> final String[] containerNodes = { "node1" };
>
> LOG.info("Trying to request node: " + containerNodes[0]);
> ContainerRequest request = new
> ContainerRequest(capability,
> containerNodes, null,
> pri, numContainers);
>
> And while the log output shows this code is being executed
> ("Trying to
> request..."), it doesn't seem to use it. Is this
> functionality that isn't
> complete yet in the 2.0.3 yarn alpha? Or am I doing
> something wrong?
>
> Thanks,
> Brian
>
>
>
>
>
>
Re: YARN - specify hosts in ContainerRequest
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
I get a container, but not on the node I'm asking for.
Thanks,
Brian
On 04/12/2013 03:01 PM, Sandy Ryza wrote:
> What do you mean when you say it doesn't seem to use the code? That
> you're not getting containers back?
>
> -Sandy
>
> On Fri, Apr 12, 2013 at 7:44 AM, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> Yes, this is exactly the DistributedShell example in the 2.0.3
> source (except for the added specification of containerNode).
>
> So, it's doing the following:
> ContainerRequest containerAsk =
> setupContainerAskForRM(askCount);
> resourceManager.addContainerRequest(containerAsk);
>
>
> // Send the request to RM
> LOG.info("Asking RM for containers" + ", askCount=" +
> askCount);
> AMResponse amResp = sendContainerAskToRM();
>
> FYI - the sendContainerAskToRM calls the allocate(progress).
>
> I don't see any NPEs in the resourcemanager's logs.
>
> Regards,
> Brian
>
>
> On 04/11/2013 02:50 PM, Harsh J wrote:
>
> Hi Brian,
>
> Your way of creation seems fine. Are you "adding" this to the
> AMRMClient somewhere though, via [1], right before you call
> allocate(progress)? Cause otherwise this object's not used at all.
>
> P.s. I'm not sure we'll handle that null for racks all that
> well in
> certain places yet; I've noticed errors in my app when I've missed
> passing elements or passed them as nulls, but I didn't rebase
> it onto
> 2.0.3 yet so haven't reported anything so far - will do so
> soon. I'd
> check the RM log to see if there's any WARN logged there with
> an NPE.
>
> [1] -
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
> <http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest%28org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest%29>
>
> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
> <bhuffman@etinternational.com
> <ma...@etinternational.com>> wrote:
>
> All,
>
> I'm starting to work with YARN and I'm trying to make some
> changes to the
> distributedshell example.
>
> Specifically I'm trying to set the host where the
> distributed shell
> container will run. I added the following code into
> ApplicationMaster.java:
> final String[] containerNodes = { "node1" };
>
> LOG.info("Trying to request node: " + containerNodes[0]);
> ContainerRequest request = new
> ContainerRequest(capability,
> containerNodes, null,
> pri, numContainers);
>
> And while the log output shows this code is being executed
> ("Trying to
> request..."), it doesn't seem to use it. Is this
> functionality that isn't
> complete yet in the 2.0.3 yarn alpha? Or am I doing
> something wrong?
>
> Thanks,
> Brian
>
>
>
>
>
>
Re: YARN - specify hosts in ContainerRequest
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
I get a container, but not on the node I'm asking for.
Thanks,
Brian
On 04/12/2013 03:01 PM, Sandy Ryza wrote:
> What do you mean when you say it doesn't seem to use the code? That
> you're not getting containers back?
>
> -Sandy
>
> On Fri, Apr 12, 2013 at 7:44 AM, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> Yes, this is exactly the DistributedShell example in the 2.0.3
> source (except for the added specification of containerNode).
>
> So, it's doing the following:
> ContainerRequest containerAsk =
> setupContainerAskForRM(askCount);
> resourceManager.addContainerRequest(containerAsk);
>
>
> // Send the request to RM
> LOG.info("Asking RM for containers" + ", askCount=" +
> askCount);
> AMResponse amResp = sendContainerAskToRM();
>
> FYI - the sendContainerAskToRM calls the allocate(progress).
>
> I don't see any NPEs in the resourcemanager's logs.
>
> Regards,
> Brian
>
>
> On 04/11/2013 02:50 PM, Harsh J wrote:
>
> Hi Brian,
>
> Your way of creation seems fine. Are you "adding" this to the
> AMRMClient somewhere though, via [1], right before you call
> allocate(progress)? Cause otherwise this object's not used at all.
>
> P.s. I'm not sure we'll handle that null for racks all that
> well in
> certain places yet; I've noticed errors in my app when I've missed
> passing elements or passed them as nulls, but I didn't rebase
> it onto
> 2.0.3 yet so haven't reported anything so far - will do so
> soon. I'd
> check the RM log to see if there's any WARN logged there with
> an NPE.
>
> [1] -
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
> <http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest%28org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest%29>
>
> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
> <bhuffman@etinternational.com
> <ma...@etinternational.com>> wrote:
>
> All,
>
> I'm starting to work with YARN and I'm trying to make some
> changes to the
> distributedshell example.
>
> Specifically I'm trying to set the host where the
> distributed shell
> container will run. I added the following code into
> ApplicationMaster.java:
> final String[] containerNodes = { "node1" };
>
> LOG.info("Trying to request node: " + containerNodes[0]);
> ContainerRequest request = new
> ContainerRequest(capability,
> containerNodes, null,
> pri, numContainers);
>
> And while the log output shows this code is being executed
> ("Trying to
> request..."), it doesn't seem to use it. Is this
> functionality that isn't
> complete yet in the 2.0.3 yarn alpha? Or am I doing
> something wrong?
>
> Thanks,
> Brian
>
>
>
>
>
>
Re: YARN - specify hosts in ContainerRequest
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
I get a container, but not on the node I'm asking for.
Thanks,
Brian
On 04/12/2013 03:01 PM, Sandy Ryza wrote:
> What do you mean when you say it doesn't seem to use the code? That
> you're not getting containers back?
>
> -Sandy
>
> On Fri, Apr 12, 2013 at 7:44 AM, Brian C. Huffman
> <bhuffman@etinternational.com <ma...@etinternational.com>>
> wrote:
>
> Yes, this is exactly the DistributedShell example in the 2.0.3
> source (except for the added specification of containerNode).
>
> So, it's doing the following:
> ContainerRequest containerAsk =
> setupContainerAskForRM(askCount);
> resourceManager.addContainerRequest(containerAsk);
>
>
> // Send the request to RM
> LOG.info("Asking RM for containers" + ", askCount=" +
> askCount);
> AMResponse amResp = sendContainerAskToRM();
>
> FYI - the sendContainerAskToRM calls the allocate(progress).
>
> I don't see any NPEs in the resourcemanager's logs.
>
> Regards,
> Brian
>
>
> On 04/11/2013 02:50 PM, Harsh J wrote:
>
> Hi Brian,
>
> Your way of creation seems fine. Are you "adding" this to the
> AMRMClient somewhere though, via [1], right before you call
> allocate(progress)? Cause otherwise this object's not used at all.
>
> P.s. I'm not sure we'll handle that null for racks all that
> well in
> certain places yet; I've noticed errors in my app when I've missed
> passing elements or passed them as nulls, but I didn't rebase
> it onto
> 2.0.3 yet so haven't reported anything so far - will do so
> soon. I'd
> check the RM log to see if there's any WARN logged there with
> an NPE.
>
> [1] -
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
> <http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest%28org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest%29>
>
> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
> <bhuffman@etinternational.com
> <ma...@etinternational.com>> wrote:
>
> All,
>
> I'm starting to work with YARN and I'm trying to make some
> changes to the
> distributedshell example.
>
> Specifically I'm trying to set the host where the
> distributed shell
> container will run. I added the following code into
> ApplicationMaster.java:
> final String[] containerNodes = { "node1" };
>
> LOG.info("Trying to request node: " + containerNodes[0]);
> ContainerRequest request = new
> ContainerRequest(capability,
> containerNodes, null,
> pri, numContainers);
>
> And while the log output shows this code is being executed
> ("Trying to
> request..."), it doesn't seem to use it. Is this
> functionality that isn't
> complete yet in the 2.0.3 yarn alpha? Or am I doing
> something wrong?
>
> Thanks,
> Brian
>
>
>
>
>
>
Re: YARN - specify hosts in ContainerRequest
Posted by Sandy Ryza <sa...@cloudera.com>.
What do you mean when you say it doesn't seem to use the code? That you're
not getting containers back?
-Sandy
On Fri, Apr 12, 2013 at 7:44 AM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> Yes, this is exactly the DistributedShell example in the 2.0.3 source
> (except for the added specification of containerNode).
>
> So, it's doing the following:
> ContainerRequest containerAsk = setupContainerAskForRM(**
> askCount);
> resourceManager.**addContainerRequest(**containerAsk);
>
>
> // Send the request to RM
> LOG.info("Asking RM for containers" + ", askCount=" + askCount);
> AMResponse amResp = sendContainerAskToRM();
>
> FYI - the sendContainerAskToRM calls the allocate(progress).
>
> I don't see any NPEs in the resourcemanager's logs.
>
> Regards,
> Brian
>
>
> On 04/11/2013 02:50 PM, Harsh J wrote:
>
>> Hi Brian,
>>
>> Your way of creation seems fine. Are you "adding" this to the
>> AMRMClient somewhere though, via [1], right before you call
>> allocate(progress)? Cause otherwise this object's not used at all.
>>
>> P.s. I'm not sure we'll handle that null for racks all that well in
>> certain places yet; I've noticed errors in my app when I've missed
>> passing elements or passed them as nulls, but I didn't rebase it onto
>> 2.0.3 yet so haven't reported anything so far - will do so soon. I'd
>> check the RM log to see if there's any WARN logged there with an NPE.
>>
>> [1] - http://hadoop.apache.org/docs/**current/api/org/apache/hadoop/**
>> yarn/client/AMRMClient.html#**addContainerRequest(org.**
>> apache.hadoop.yarn.client.**AMRMClient.ContainerRequest)<http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)>
>>
>> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
>> <bh...@etinternational.com> wrote:
>>
>>> All,
>>>
>>> I'm starting to work with YARN and I'm trying to make some changes to the
>>> distributedshell example.
>>>
>>> Specifically I'm trying to set the host where the distributed shell
>>> container will run. I added the following code into
>>> ApplicationMaster.java:
>>> final String[] containerNodes = { "node1" };
>>>
>>> LOG.info("Trying to request node: " + containerNodes[0]);
>>> ContainerRequest request = new ContainerRequest(capability,
>>> containerNodes, null,
>>> pri, numContainers);
>>>
>>> And while the log output shows this code is being executed ("Trying to
>>> request..."), it doesn't seem to use it. Is this functionality that
>>> isn't
>>> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>>>
>>> Thanks,
>>> Brian
>>>
>>>
>>
>>
>
>
Re: YARN - specify hosts in ContainerRequest
Posted by Sandy Ryza <sa...@cloudera.com>.
What do you mean when you say it doesn't seem to use the code? That you're
not getting containers back?
-Sandy
On Fri, Apr 12, 2013 at 7:44 AM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> Yes, this is exactly the DistributedShell example in the 2.0.3 source
> (except for the added specification of containerNode).
>
> So, it's doing the following:
> ContainerRequest containerAsk = setupContainerAskForRM(**
> askCount);
> resourceManager.**addContainerRequest(**containerAsk);
>
>
> // Send the request to RM
> LOG.info("Asking RM for containers" + ", askCount=" + askCount);
> AMResponse amResp = sendContainerAskToRM();
>
> FYI - the sendContainerAskToRM calls the allocate(progress).
>
> I don't see any NPEs in the resourcemanager's logs.
>
> Regards,
> Brian
>
>
> On 04/11/2013 02:50 PM, Harsh J wrote:
>
>> Hi Brian,
>>
>> Your way of creation seems fine. Are you "adding" this to the
>> AMRMClient somewhere though, via [1], right before you call
>> allocate(progress)? Cause otherwise this object's not used at all.
>>
>> P.s. I'm not sure we'll handle that null for racks all that well in
>> certain places yet; I've noticed errors in my app when I've missed
>> passing elements or passed them as nulls, but I didn't rebase it onto
>> 2.0.3 yet so haven't reported anything so far - will do so soon. I'd
>> check the RM log to see if there's any WARN logged there with an NPE.
>>
>> [1] - http://hadoop.apache.org/docs/**current/api/org/apache/hadoop/**
>> yarn/client/AMRMClient.html#**addContainerRequest(org.**
>> apache.hadoop.yarn.client.**AMRMClient.ContainerRequest)<http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)>
>>
>> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
>> <bh...@etinternational.com> wrote:
>>
>>> All,
>>>
>>> I'm starting to work with YARN and I'm trying to make some changes to the
>>> distributedshell example.
>>>
>>> Specifically I'm trying to set the host where the distributed shell
>>> container will run. I added the following code into
>>> ApplicationMaster.java:
>>> final String[] containerNodes = { "node1" };
>>>
>>> LOG.info("Trying to request node: " + containerNodes[0]);
>>> ContainerRequest request = new ContainerRequest(capability,
>>> containerNodes, null,
>>> pri, numContainers);
>>>
>>> And while the log output shows this code is being executed ("Trying to
>>> request..."), it doesn't seem to use it. Is this functionality that
>>> isn't
>>> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>>>
>>> Thanks,
>>> Brian
>>>
>>>
>>
>>
>
>
Re: YARN - specify hosts in ContainerRequest
Posted by Sandy Ryza <sa...@cloudera.com>.
What do you mean when you say it doesn't seem to use the code? That you're
not getting containers back?
-Sandy
On Fri, Apr 12, 2013 at 7:44 AM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> Yes, this is exactly the DistributedShell example in the 2.0.3 source
> (except for the added specification of containerNode).
>
> So, it's doing the following:
> ContainerRequest containerAsk = setupContainerAskForRM(**
> askCount);
> resourceManager.**addContainerRequest(**containerAsk);
>
>
> // Send the request to RM
> LOG.info("Asking RM for containers" + ", askCount=" + askCount);
> AMResponse amResp = sendContainerAskToRM();
>
> FYI - the sendContainerAskToRM calls the allocate(progress).
>
> I don't see any NPEs in the resourcemanager's logs.
>
> Regards,
> Brian
>
>
> On 04/11/2013 02:50 PM, Harsh J wrote:
>
>> Hi Brian,
>>
>> Your way of creation seems fine. Are you "adding" this to the
>> AMRMClient somewhere though, via [1], right before you call
>> allocate(progress)? Cause otherwise this object's not used at all.
>>
>> P.s. I'm not sure we'll handle that null for racks all that well in
>> certain places yet; I've noticed errors in my app when I've missed
>> passing elements or passed them as nulls, but I didn't rebase it onto
>> 2.0.3 yet so haven't reported anything so far - will do so soon. I'd
>> check the RM log to see if there's any WARN logged there with an NPE.
>>
>> [1] - http://hadoop.apache.org/docs/**current/api/org/apache/hadoop/**
>> yarn/client/AMRMClient.html#**addContainerRequest(org.**
>> apache.hadoop.yarn.client.**AMRMClient.ContainerRequest)<http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)>
>>
>> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
>> <bh...@etinternational.com> wrote:
>>
>>> All,
>>>
>>> I'm starting to work with YARN and I'm trying to make some changes to the
>>> distributedshell example.
>>>
>>> Specifically I'm trying to set the host where the distributed shell
>>> container will run. I added the following code into
>>> ApplicationMaster.java:
>>> final String[] containerNodes = { "node1" };
>>>
>>> LOG.info("Trying to request node: " + containerNodes[0]);
>>> ContainerRequest request = new ContainerRequest(capability,
>>> containerNodes, null,
>>> pri, numContainers);
>>>
>>> And while the log output shows this code is being executed ("Trying to
>>> request..."), it doesn't seem to use it. Is this functionality that
>>> isn't
>>> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>>>
>>> Thanks,
>>> Brian
>>>
>>>
>>
>>
>
>
Re: YARN - specify hosts in ContainerRequest
Posted by Sandy Ryza <sa...@cloudera.com>.
What do you mean when you say it doesn't seem to use the code? That you're
not getting containers back?
-Sandy
On Fri, Apr 12, 2013 at 7:44 AM, Brian C. Huffman <
bhuffman@etinternational.com> wrote:
> Yes, this is exactly the DistributedShell example in the 2.0.3 source
> (except for the added specification of containerNode).
>
> So, it's doing the following:
> ContainerRequest containerAsk = setupContainerAskForRM(**
> askCount);
> resourceManager.**addContainerRequest(**containerAsk);
>
>
> // Send the request to RM
> LOG.info("Asking RM for containers" + ", askCount=" + askCount);
> AMResponse amResp = sendContainerAskToRM();
>
> FYI - the sendContainerAskToRM calls the allocate(progress).
>
> I don't see any NPEs in the resourcemanager's logs.
>
> Regards,
> Brian
>
>
> On 04/11/2013 02:50 PM, Harsh J wrote:
>
>> Hi Brian,
>>
>> Your way of creation seems fine. Are you "adding" this to the
>> AMRMClient somewhere though, via [1], right before you call
>> allocate(progress)? Cause otherwise this object's not used at all.
>>
>> P.s. I'm not sure we'll handle that null for racks all that well in
>> certain places yet; I've noticed errors in my app when I've missed
>> passing elements or passed them as nulls, but I didn't rebase it onto
>> 2.0.3 yet so haven't reported anything so far - will do so soon. I'd
>> check the RM log to see if there's any WARN logged there with an NPE.
>>
>> [1] - http://hadoop.apache.org/docs/**current/api/org/apache/hadoop/**
>> yarn/client/AMRMClient.html#**addContainerRequest(org.**
>> apache.hadoop.yarn.client.**AMRMClient.ContainerRequest)<http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)>
>>
>> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
>> <bh...@etinternational.com> wrote:
>>
>>> All,
>>>
>>> I'm starting to work with YARN and I'm trying to make some changes to the
>>> distributedshell example.
>>>
>>> Specifically I'm trying to set the host where the distributed shell
>>> container will run. I added the following code into
>>> ApplicationMaster.java:
>>> final String[] containerNodes = { "node1" };
>>>
>>> LOG.info("Trying to request node: " + containerNodes[0]);
>>> ContainerRequest request = new ContainerRequest(capability,
>>> containerNodes, null,
>>> pri, numContainers);
>>>
>>> And while the log output shows this code is being executed ("Trying to
>>> request..."), it doesn't seem to use it. Is this functionality that
>>> isn't
>>> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>>>
>>> Thanks,
>>> Brian
>>>
>>>
>>
>>
>
>
Re: YARN - specify hosts in ContainerRequest
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Yes, this is exactly the DistributedShell example in the 2.0.3 source
(except for the added specification of containerNode).
So, it's doing the following:
ContainerRequest containerAsk = setupContainerAskForRM(askCount);
resourceManager.addContainerRequest(containerAsk);
// Send the request to RM
LOG.info("Asking RM for containers" + ", askCount=" + askCount);
AMResponse amResp = sendContainerAskToRM();
FYI - the sendContainerAskToRM calls the allocate(progress).
I don't see any NPEs in the resourcemanager's logs.
Regards,
Brian
On 04/11/2013 02:50 PM, Harsh J wrote:
> Hi Brian,
>
> Your way of creation seems fine. Are you "adding" this to the
> AMRMClient somewhere though, via [1], right before you call
> allocate(progress)? Cause otherwise this object's not used at all.
>
> P.s. I'm not sure we'll handle that null for racks all that well in
> certain places yet; I've noticed errors in my app when I've missed
> passing elements or passed them as nulls, but I didn't rebase it onto
> 2.0.3 yet so haven't reported anything so far - will do so soon. I'd
> check the RM log to see if there's any WARN logged there with an NPE.
>
> [1] - http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
>
> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
> <bh...@etinternational.com> wrote:
>> All,
>>
>> I'm starting to work with YARN and I'm trying to make some changes to the
>> distributedshell example.
>>
>> Specifically I'm trying to set the host where the distributed shell
>> container will run. I added the following code into ApplicationMaster.java:
>> final String[] containerNodes = { "node1" };
>>
>> LOG.info("Trying to request node: " + containerNodes[0]);
>> ContainerRequest request = new ContainerRequest(capability,
>> containerNodes, null,
>> pri, numContainers);
>>
>> And while the log output shows this code is being executed ("Trying to
>> request..."), it doesn't seem to use it. Is this functionality that isn't
>> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>>
>> Thanks,
>> Brian
>>
>
>
Re: YARN - specify hosts in ContainerRequest
Posted by Arun C Murthy <ac...@hortonworks.com>.
There shouldn't be issues handling nulls etc. - obviously depends on scheduler, but if you see any we will fix it ASAP.
On Apr 12, 2013, at 12:20 AM, Harsh J wrote:
> Hi Brian,
>
> Your way of creation seems fine. Are you "adding" this to the
> AMRMClient somewhere though, via [1], right before you call
> allocate(progress)? Cause otherwise this object's not used at all.
>
> P.s. I'm not sure we'll handle that null for racks all that well in
> certain places yet; I've noticed errors in my app when I've missed
> passing elements or passed them as nulls, but I didn't rebase it onto
> 2.0.3 yet so haven't reported anything so far - will do so soon. I'd
> check the RM log to see if there's any WARN logged there with an NPE.
>
> [1] - http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
>
> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
> <bh...@etinternational.com> wrote:
>> All,
>>
>> I'm starting to work with YARN and I'm trying to make some changes to the
>> distributedshell example.
>>
>> Specifically I'm trying to set the host where the distributed shell
>> container will run. I added the following code into ApplicationMaster.java:
>> final String[] containerNodes = { "node1" };
>>
>> LOG.info("Trying to request node: " + containerNodes[0]);
>> ContainerRequest request = new ContainerRequest(capability,
>> containerNodes, null,
>> pri, numContainers);
>>
>> And while the log output shows this code is being executed ("Trying to
>> request..."), it doesn't seem to use it. Is this functionality that isn't
>> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>>
>> Thanks,
>> Brian
>>
>
>
>
> --
> Harsh J
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: YARN - specify hosts in ContainerRequest
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Yes, this is exactly the DistributedShell example in the 2.0.3 source
(except for the added specification of containerNode).
So, it's doing the following:
ContainerRequest containerAsk = setupContainerAskForRM(askCount);
resourceManager.addContainerRequest(containerAsk);
// Send the request to RM
LOG.info("Asking RM for containers" + ", askCount=" + askCount);
AMResponse amResp = sendContainerAskToRM();
FYI - the sendContainerAskToRM calls the allocate(progress).
I don't see any NPEs in the resourcemanager's logs.
Regards,
Brian
On 04/11/2013 02:50 PM, Harsh J wrote:
> Hi Brian,
>
> Your way of creation seems fine. Are you "adding" this to the
> AMRMClient somewhere though, via [1], right before you call
> allocate(progress)? Cause otherwise this object's not used at all.
>
> P.s. I'm not sure we'll handle that null for racks all that well in
> certain places yet; I've noticed errors in my app when I've missed
> passing elements or passed them as nulls, but I didn't rebase it onto
> 2.0.3 yet so haven't reported anything so far - will do so soon. I'd
> check the RM log to see if there's any WARN logged there with an NPE.
>
> [1] - http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
>
> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
> <bh...@etinternational.com> wrote:
>> All,
>>
>> I'm starting to work with YARN and I'm trying to make some changes to the
>> distributedshell example.
>>
>> Specifically I'm trying to set the host where the distributed shell
>> container will run. I added the following code into ApplicationMaster.java:
>> final String[] containerNodes = { "node1" };
>>
>> LOG.info("Trying to request node: " + containerNodes[0]);
>> ContainerRequest request = new ContainerRequest(capability,
>> containerNodes, null,
>> pri, numContainers);
>>
>> And while the log output shows this code is being executed ("Trying to
>> request..."), it doesn't seem to use it. Is this functionality that isn't
>> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>>
>> Thanks,
>> Brian
>>
>
>
Re: YARN - specify hosts in ContainerRequest
Posted by Arun C Murthy <ac...@hortonworks.com>.
There shouldn't be issues handling nulls etc. - obviously depends on scheduler, but if you see any we will fix it ASAP.
On Apr 12, 2013, at 12:20 AM, Harsh J wrote:
> Hi Brian,
>
> Your way of creation seems fine. Are you "adding" this to the
> AMRMClient somewhere though, via [1], right before you call
> allocate(progress)? Cause otherwise this object's not used at all.
>
> P.s. I'm not sure we'll handle that null for racks all that well in
> certain places yet; I've noticed errors in my app when I've missed
> passing elements or passed them as nulls, but I didn't rebase it onto
> 2.0.3 yet so haven't reported anything so far - will do so soon. I'd
> check the RM log to see if there's any WARN logged there with an NPE.
>
> [1] - http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
>
> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
> <bh...@etinternational.com> wrote:
>> All,
>>
>> I'm starting to work with YARN and I'm trying to make some changes to the
>> distributedshell example.
>>
>> Specifically I'm trying to set the host where the distributed shell
>> container will run. I added the following code into ApplicationMaster.java:
>> final String[] containerNodes = { "node1" };
>>
>> LOG.info("Trying to request node: " + containerNodes[0]);
>> ContainerRequest request = new ContainerRequest(capability,
>> containerNodes, null,
>> pri, numContainers);
>>
>> And while the log output shows this code is being executed ("Trying to
>> request..."), it doesn't seem to use it. Is this functionality that isn't
>> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>>
>> Thanks,
>> Brian
>>
>
>
>
> --
> Harsh J
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: YARN - specify hosts in ContainerRequest
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Yes, this is exactly the DistributedShell example in the 2.0.3 source
(except for the added specification of containerNode).
So, it's doing the following:
ContainerRequest containerAsk = setupContainerAskForRM(askCount);
resourceManager.addContainerRequest(containerAsk);
// Send the request to RM
LOG.info("Asking RM for containers" + ", askCount=" + askCount);
AMResponse amResp = sendContainerAskToRM();
FYI - the sendContainerAskToRM calls the allocate(progress).
I don't see any NPEs in the resourcemanager's logs.
Regards,
Brian
On 04/11/2013 02:50 PM, Harsh J wrote:
> Hi Brian,
>
> Your way of creation seems fine. Are you "adding" this to the
> AMRMClient somewhere though, via [1], right before you call
> allocate(progress)? Cause otherwise this object's not used at all.
>
> P.s. I'm not sure we'll handle that null for racks all that well in
> certain places yet; I've noticed errors in my app when I've missed
> passing elements or passed them as nulls, but I didn't rebase it onto
> 2.0.3 yet so haven't reported anything so far - will do so soon. I'd
> check the RM log to see if there's any WARN logged there with an NPE.
>
> [1] - http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
>
> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
> <bh...@etinternational.com> wrote:
>> All,
>>
>> I'm starting to work with YARN and I'm trying to make some changes to the
>> distributedshell example.
>>
>> Specifically I'm trying to set the host where the distributed shell
>> container will run. I added the following code into ApplicationMaster.java:
>> final String[] containerNodes = { "node1" };
>>
>> LOG.info("Trying to request node: " + containerNodes[0]);
>> ContainerRequest request = new ContainerRequest(capability,
>> containerNodes, null,
>> pri, numContainers);
>>
>> And while the log output shows this code is being executed ("Trying to
>> request..."), it doesn't seem to use it. Is this functionality that isn't
>> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>>
>> Thanks,
>> Brian
>>
>
>
Re: YARN - specify hosts in ContainerRequest
Posted by Arun C Murthy <ac...@hortonworks.com>.
There shouldn't be issues handling nulls etc. - obviously depends on scheduler, but if you see any we will fix it ASAP.
On Apr 12, 2013, at 12:20 AM, Harsh J wrote:
> Hi Brian,
>
> Your way of creation seems fine. Are you "adding" this to the
> AMRMClient somewhere though, via [1], right before you call
> allocate(progress)? Cause otherwise this object's not used at all.
>
> P.s. I'm not sure we'll handle that null for racks all that well in
> certain places yet; I've noticed errors in my app when I've missed
> passing elements or passed them as nulls, but I didn't rebase it onto
> 2.0.3 yet so haven't reported anything so far - will do so soon. I'd
> check the RM log to see if there's any WARN logged there with an NPE.
>
> [1] - http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
>
> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
> <bh...@etinternational.com> wrote:
>> All,
>>
>> I'm starting to work with YARN and I'm trying to make some changes to the
>> distributedshell example.
>>
>> Specifically I'm trying to set the host where the distributed shell
>> container will run. I added the following code into ApplicationMaster.java:
>> final String[] containerNodes = { "node1" };
>>
>> LOG.info("Trying to request node: " + containerNodes[0]);
>> ContainerRequest request = new ContainerRequest(capability,
>> containerNodes, null,
>> pri, numContainers);
>>
>> And while the log output shows this code is being executed ("Trying to
>> request..."), it doesn't seem to use it. Is this functionality that isn't
>> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>>
>> Thanks,
>> Brian
>>
>
>
>
> --
> Harsh J
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: YARN - specify hosts in ContainerRequest
Posted by "Brian C. Huffman" <bh...@etinternational.com>.
Yes, this is exactly the DistributedShell example in the 2.0.3 source
(except for the added specification of containerNode).
So, it's doing the following:
ContainerRequest containerAsk = setupContainerAskForRM(askCount);
resourceManager.addContainerRequest(containerAsk);
// Send the request to RM
LOG.info("Asking RM for containers" + ", askCount=" + askCount);
AMResponse amResp = sendContainerAskToRM();
FYI - the sendContainerAskToRM calls the allocate(progress).
I don't see any NPEs in the resourcemanager's logs.
Regards,
Brian
On 04/11/2013 02:50 PM, Harsh J wrote:
> Hi Brian,
>
> Your way of creation seems fine. Are you "adding" this to the
> AMRMClient somewhere though, via [1], right before you call
> allocate(progress)? Cause otherwise this object's not used at all.
>
> P.s. I'm not sure we'll handle that null for racks all that well in
> certain places yet; I've noticed errors in my app when I've missed
> passing elements or passed them as nulls, but I didn't rebase it onto
> 2.0.3 yet so haven't reported anything so far - will do so soon. I'd
> check the RM log to see if there's any WARN logged there with an NPE.
>
> [1] - http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
>
> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
> <bh...@etinternational.com> wrote:
>> All,
>>
>> I'm starting to work with YARN and I'm trying to make some changes to the
>> distributedshell example.
>>
>> Specifically I'm trying to set the host where the distributed shell
>> container will run. I added the following code into ApplicationMaster.java:
>> final String[] containerNodes = { "node1" };
>>
>> LOG.info("Trying to request node: " + containerNodes[0]);
>> ContainerRequest request = new ContainerRequest(capability,
>> containerNodes, null,
>> pri, numContainers);
>>
>> And while the log output shows this code is being executed ("Trying to
>> request..."), it doesn't seem to use it. Is this functionality that isn't
>> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>>
>> Thanks,
>> Brian
>>
>
>
Re: YARN - specify hosts in ContainerRequest
Posted by Arun C Murthy <ac...@hortonworks.com>.
There shouldn't be issues handling nulls etc. - obviously depends on scheduler, but if you see any we will fix it ASAP.
On Apr 12, 2013, at 12:20 AM, Harsh J wrote:
> Hi Brian,
>
> Your way of creation seems fine. Are you "adding" this to the
> AMRMClient somewhere though, via [1], right before you call
> allocate(progress)? Cause otherwise this object's not used at all.
>
> P.s. I'm not sure we'll handle that null for racks all that well in
> certain places yet; I've noticed errors in my app when I've missed
> passing elements or passed them as nulls, but I didn't rebase it onto
> 2.0.3 yet so haven't reported anything so far - will do so soon. I'd
> check the RM log to see if there's any WARN logged there with an NPE.
>
> [1] - http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
>
> On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
> <bh...@etinternational.com> wrote:
>> All,
>>
>> I'm starting to work with YARN and I'm trying to make some changes to the
>> distributedshell example.
>>
>> Specifically I'm trying to set the host where the distributed shell
>> container will run. I added the following code into ApplicationMaster.java:
>> final String[] containerNodes = { "node1" };
>>
>> LOG.info("Trying to request node: " + containerNodes[0]);
>> ContainerRequest request = new ContainerRequest(capability,
>> containerNodes, null,
>> pri, numContainers);
>>
>> And while the log output shows this code is being executed ("Trying to
>> request..."), it doesn't seem to use it. Is this functionality that isn't
>> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>>
>> Thanks,
>> Brian
>>
>
>
>
> --
> Harsh J
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: YARN - specify hosts in ContainerRequest
Posted by Harsh J <ha...@cloudera.com>.
Hi Brian,
Your way of creation seems fine. Are you "adding" this to the
AMRMClient somewhere though, via [1], right before you call
allocate(progress)? Cause otherwise this object's not used at all.
P.s. I'm not sure we'll handle that null for racks all that well in
certain places yet; I've noticed errors in my app when I've missed
passing elements or passed them as nulls, but I didn't rebase it onto
2.0.3 yet so haven't reported anything so far - will do so soon. I'd
check the RM log to see if there's any WARN logged there with an NPE.
[1] - http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
<bh...@etinternational.com> wrote:
> All,
>
> I'm starting to work with YARN and I'm trying to make some changes to the
> distributedshell example.
>
> Specifically I'm trying to set the host where the distributed shell
> container will run. I added the following code into ApplicationMaster.java:
> final String[] containerNodes = { "node1" };
>
> LOG.info("Trying to request node: " + containerNodes[0]);
> ContainerRequest request = new ContainerRequest(capability,
> containerNodes, null,
> pri, numContainers);
>
> And while the log output shows this code is being executed ("Trying to
> request..."), it doesn't seem to use it. Is this functionality that isn't
> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>
> Thanks,
> Brian
>
--
Harsh J
Re: YARN - specify hosts in ContainerRequest
Posted by Harsh J <ha...@cloudera.com>.
Hi Brian,
Your way of creation seems fine. Are you "adding" this to the
AMRMClient somewhere though, via [1], right before you call
allocate(progress)? Cause otherwise this object's not used at all.
P.s. I'm not sure we'll handle that null for racks all that well in
certain places yet; I've noticed errors in my app when I've missed
passing elements or passed them as nulls, but I didn't rebase it onto
2.0.3 yet so haven't reported anything so far - will do so soon. I'd
check the RM log to see if there's any WARN logged there with an NPE.
[1] - http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
<bh...@etinternational.com> wrote:
> All,
>
> I'm starting to work with YARN and I'm trying to make some changes to the
> distributedshell example.
>
> Specifically I'm trying to set the host where the distributed shell
> container will run. I added the following code into ApplicationMaster.java:
> final String[] containerNodes = { "node1" };
>
> LOG.info("Trying to request node: " + containerNodes[0]);
> ContainerRequest request = new ContainerRequest(capability,
> containerNodes, null,
> pri, numContainers);
>
> And while the log output shows this code is being executed ("Trying to
> request..."), it doesn't seem to use it. Is this functionality that isn't
> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>
> Thanks,
> Brian
>
--
Harsh J
Re: YARN - specify hosts in ContainerRequest
Posted by Harsh J <ha...@cloudera.com>.
Hi Brian,
Your way of creation seems fine. Are you "adding" this to the
AMRMClient somewhere though, via [1], right before you call
allocate(progress)? Cause otherwise this object's not used at all.
P.s. I'm not sure we'll handle that null for racks all that well in
certain places yet; I've noticed errors in my app when I've missed
passing elements or passed them as nulls, but I didn't rebase it onto
2.0.3 yet so haven't reported anything so far - will do so soon. I'd
check the RM log to see if there's any WARN logged there with an NPE.
[1] - http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
<bh...@etinternational.com> wrote:
> All,
>
> I'm starting to work with YARN and I'm trying to make some changes to the
> distributedshell example.
>
> Specifically I'm trying to set the host where the distributed shell
> container will run. I added the following code into ApplicationMaster.java:
> final String[] containerNodes = { "node1" };
>
> LOG.info("Trying to request node: " + containerNodes[0]);
> ContainerRequest request = new ContainerRequest(capability,
> containerNodes, null,
> pri, numContainers);
>
> And while the log output shows this code is being executed ("Trying to
> request..."), it doesn't seem to use it. Is this functionality that isn't
> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>
> Thanks,
> Brian
>
--
Harsh J
Re: YARN - specify hosts in ContainerRequest
Posted by Harsh J <ha...@cloudera.com>.
Hi Brian,
Your way of creation seems fine. Are you "adding" this to the
AMRMClient somewhere though, via [1], right before you call
allocate(progress)? Cause otherwise this object's not used at all.
P.s. I'm not sure we'll handle that null for racks all that well in
certain places yet; I've noticed errors in my app when I've missed
passing elements or passed them as nulls, but I didn't rebase it onto
2.0.3 yet so haven't reported anything so far - will do so soon. I'd
check the RM log to see if there's any WARN logged there with an NPE.
[1] - http://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/AMRMClient.html#addContainerRequest(org.apache.hadoop.yarn.client.AMRMClient.ContainerRequest)
On Thu, Apr 11, 2013 at 11:30 PM, Brian C. Huffman
<bh...@etinternational.com> wrote:
> All,
>
> I'm starting to work with YARN and I'm trying to make some changes to the
> distributedshell example.
>
> Specifically I'm trying to set the host where the distributed shell
> container will run. I added the following code into ApplicationMaster.java:
> final String[] containerNodes = { "node1" };
>
> LOG.info("Trying to request node: " + containerNodes[0]);
> ContainerRequest request = new ContainerRequest(capability,
> containerNodes, null,
> pri, numContainers);
>
> And while the log output shows this code is being executed ("Trying to
> request..."), it doesn't seem to use it. Is this functionality that isn't
> complete yet in the 2.0.3 yarn alpha? Or am I doing something wrong?
>
> Thanks,
> Brian
>
--
Harsh J