You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Rahul Bhattacharjee <ra...@gmail.com> on 2013/09/05 17:03:48 UTC
Question related to resource allocation in Yarn!
Hi,
I am trying to make a small poc on top of yarn.
Within the launched application master , I am trying to request for 50
containers and launch a same task on those allocated containers.
My config : AM registration response minimumCapability {, memory: 1024,
virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32,
},
1) I am asking for 1G mem and 1 core container to the RM. Ideally the RM
should return me 6 - 7 containers , but the response always returns with
only 2 containers.
Why is that ?
2) So , when in the first ask 2 containers are returned , then I again
required the RM for 50 - 2 = 48 containers. I keep getting 0 containers ,
even if the previously started containers have finished.
Why is that ?
Any link explaining the allocate request of RM would be very helpful.
Thanks,
Rahul
Re: Question related to resource allocation in Yarn!
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
Hi Devaraj,
I am on Hadoop 2.0.4 . I am able to get containers now and my yarn app runs
properly. I am setting hostname as * , while requesting containers.
There is no problem as of now , only thing is I am allocated only 2
containers at one time , however I believe that the node manager can run
more containers.
2013-09-06 09:53:38,433 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode:
Assigned container container_1378441324025_0001_01_000001 of capacity
<memory:100, vCores:1> on host storyacid-lm:55407, which currently has 1
containers, <memory:100, vCores:1> used and <memory:8092, vCores:15>
available
I am requesting containers with 100 mb mem and 1 core. If I know more
about how is the capacity if calculated per node , or how the allocation is
done , then it would be useful.
Thanks for the help!
Rahul
On Fri, Sep 6, 2013 at 10:31 AM, Devaraj k <de...@huawei.com> wrote:
> Hi Rahul,****
>
> ** **
>
> Could you tell me, what is the version you are using?****
>
> ****
>
> **· **If you want a container, you need to issue 3 resource
> requests (1-node local, 1-rack local and 1-Any(*) ). If you are using
> 2.1.0-beta<https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=ASC&sorter/field=priority&pid=12313722&customfield_12310320=12324029>or later versions, you can set the Relax Locality flag to false for getting
> only on the specified host.****
>
> Can you also share the code how you are requesting for containers…so that
> we can help you better..****
>
> ** **
>
> Thanks****
>
> Devaraj k****
>
> ** **
>
> *From:* Rahul Bhattacharjee [mailto:rahul.rec.dgp@gmail.com]
> *Sent:* 06 September 2013 09:43
> *To:* user@hadoop.apache.org
> *Subject:* Re: Question related to resource allocation in Yarn!****
>
> ** **
>
> I could progress a bit on this.
>
> I was not setting responseId while asking for containers.****
>
> Still I have one question as why I am only been allocated two containers
> whereas node manager can run more containers.
>
> Response while registering the application master -
> AM registration response minimumCapability {, memory: 1024, virtual_cores:
> 1, }, maximumCapability {, memory: 8192, virtual_cores: 32, }, ****
>
> Thanks,****
>
> Rahul****
>
> ** **
>
> On Thu, Sep 5, 2013 at 8:33 PM, Rahul Bhattacharjee <
> rahul.rec.dgp@gmail.com> wrote:****
>
> Hi,****
>
> I am trying to make a small poc on top of yarn.****
>
> Within the launched application master , I am trying to request for 50
> containers and launch a same task on those allocated containers.****
>
> My config : AM registration response minimumCapability {, memory: 1024,
> virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32,
> }, ****
>
> 1) I am asking for 1G mem and 1 core container to the RM. Ideally the RM
> should return me 6 - 7 containers , but the response always returns with
> only 2 containers. ****
>
> Why is that ?****
>
> 2) So , when in the first ask 2 containers are returned , then I again
> required the RM for 50 - 2 = 48 containers. I keep getting 0 containers ,
> even if the previously started containers have finished.****
>
> Why is that ?****
>
> Any link explaining the allocate request of RM would be very helpful.****
>
> ** **
>
> Thanks,
> Rahul****
>
> ** **
>
Re: Question related to resource allocation in Yarn!
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
Hi Devaraj,
I am on Hadoop 2.0.4 . I am able to get containers now and my yarn app runs
properly. I am setting hostname as * , while requesting containers.
There is no problem as of now , only thing is I am allocated only 2
containers at one time , however I believe that the node manager can run
more containers.
2013-09-06 09:53:38,433 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode:
Assigned container container_1378441324025_0001_01_000001 of capacity
<memory:100, vCores:1> on host storyacid-lm:55407, which currently has 1
containers, <memory:100, vCores:1> used and <memory:8092, vCores:15>
available
I am requesting containers with 100 mb mem and 1 core. If I know more
about how is the capacity if calculated per node , or how the allocation is
done , then it would be useful.
Thanks for the help!
Rahul
On Fri, Sep 6, 2013 at 10:31 AM, Devaraj k <de...@huawei.com> wrote:
> Hi Rahul,****
>
> ** **
>
> Could you tell me, what is the version you are using?****
>
> ****
>
> **· **If you want a container, you need to issue 3 resource
> requests (1-node local, 1-rack local and 1-Any(*) ). If you are using
> 2.1.0-beta<https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=ASC&sorter/field=priority&pid=12313722&customfield_12310320=12324029>or later versions, you can set the Relax Locality flag to false for getting
> only on the specified host.****
>
> Can you also share the code how you are requesting for containers…so that
> we can help you better..****
>
> ** **
>
> Thanks****
>
> Devaraj k****
>
> ** **
>
> *From:* Rahul Bhattacharjee [mailto:rahul.rec.dgp@gmail.com]
> *Sent:* 06 September 2013 09:43
> *To:* user@hadoop.apache.org
> *Subject:* Re: Question related to resource allocation in Yarn!****
>
> ** **
>
> I could progress a bit on this.
>
> I was not setting responseId while asking for containers.****
>
> Still I have one question as why I am only been allocated two containers
> whereas node manager can run more containers.
>
> Response while registering the application master -
> AM registration response minimumCapability {, memory: 1024, virtual_cores:
> 1, }, maximumCapability {, memory: 8192, virtual_cores: 32, }, ****
>
> Thanks,****
>
> Rahul****
>
> ** **
>
> On Thu, Sep 5, 2013 at 8:33 PM, Rahul Bhattacharjee <
> rahul.rec.dgp@gmail.com> wrote:****
>
> Hi,****
>
> I am trying to make a small poc on top of yarn.****
>
> Within the launched application master , I am trying to request for 50
> containers and launch a same task on those allocated containers.****
>
> My config : AM registration response minimumCapability {, memory: 1024,
> virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32,
> }, ****
>
> 1) I am asking for 1G mem and 1 core container to the RM. Ideally the RM
> should return me 6 - 7 containers , but the response always returns with
> only 2 containers. ****
>
> Why is that ?****
>
> 2) So , when in the first ask 2 containers are returned , then I again
> required the RM for 50 - 2 = 48 containers. I keep getting 0 containers ,
> even if the previously started containers have finished.****
>
> Why is that ?****
>
> Any link explaining the allocate request of RM would be very helpful.****
>
> ** **
>
> Thanks,
> Rahul****
>
> ** **
>
Re: Question related to resource allocation in Yarn!
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
Hi Devaraj,
I am on Hadoop 2.0.4 . I am able to get containers now and my yarn app runs
properly. I am setting hostname as * , while requesting containers.
There is no problem as of now , only thing is I am allocated only 2
containers at one time , however I believe that the node manager can run
more containers.
2013-09-06 09:53:38,433 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode:
Assigned container container_1378441324025_0001_01_000001 of capacity
<memory:100, vCores:1> on host storyacid-lm:55407, which currently has 1
containers, <memory:100, vCores:1> used and <memory:8092, vCores:15>
available
I am requesting containers with 100 mb mem and 1 core. If I know more
about how is the capacity if calculated per node , or how the allocation is
done , then it would be useful.
Thanks for the help!
Rahul
On Fri, Sep 6, 2013 at 10:31 AM, Devaraj k <de...@huawei.com> wrote:
> Hi Rahul,****
>
> ** **
>
> Could you tell me, what is the version you are using?****
>
> ****
>
> **· **If you want a container, you need to issue 3 resource
> requests (1-node local, 1-rack local and 1-Any(*) ). If you are using
> 2.1.0-beta<https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=ASC&sorter/field=priority&pid=12313722&customfield_12310320=12324029>or later versions, you can set the Relax Locality flag to false for getting
> only on the specified host.****
>
> Can you also share the code how you are requesting for containers…so that
> we can help you better..****
>
> ** **
>
> Thanks****
>
> Devaraj k****
>
> ** **
>
> *From:* Rahul Bhattacharjee [mailto:rahul.rec.dgp@gmail.com]
> *Sent:* 06 September 2013 09:43
> *To:* user@hadoop.apache.org
> *Subject:* Re: Question related to resource allocation in Yarn!****
>
> ** **
>
> I could progress a bit on this.
>
> I was not setting responseId while asking for containers.****
>
> Still I have one question as why I am only been allocated two containers
> whereas node manager can run more containers.
>
> Response while registering the application master -
> AM registration response minimumCapability {, memory: 1024, virtual_cores:
> 1, }, maximumCapability {, memory: 8192, virtual_cores: 32, }, ****
>
> Thanks,****
>
> Rahul****
>
> ** **
>
> On Thu, Sep 5, 2013 at 8:33 PM, Rahul Bhattacharjee <
> rahul.rec.dgp@gmail.com> wrote:****
>
> Hi,****
>
> I am trying to make a small poc on top of yarn.****
>
> Within the launched application master , I am trying to request for 50
> containers and launch a same task on those allocated containers.****
>
> My config : AM registration response minimumCapability {, memory: 1024,
> virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32,
> }, ****
>
> 1) I am asking for 1G mem and 1 core container to the RM. Ideally the RM
> should return me 6 - 7 containers , but the response always returns with
> only 2 containers. ****
>
> Why is that ?****
>
> 2) So , when in the first ask 2 containers are returned , then I again
> required the RM for 50 - 2 = 48 containers. I keep getting 0 containers ,
> even if the previously started containers have finished.****
>
> Why is that ?****
>
> Any link explaining the allocate request of RM would be very helpful.****
>
> ** **
>
> Thanks,
> Rahul****
>
> ** **
>
Re: Question related to resource allocation in Yarn!
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
Hi Devaraj,
I am on Hadoop 2.0.4 . I am able to get containers now and my yarn app runs
properly. I am setting hostname as * , while requesting containers.
There is no problem as of now , only thing is I am allocated only 2
containers at one time , however I believe that the node manager can run
more containers.
2013-09-06 09:53:38,433 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode:
Assigned container container_1378441324025_0001_01_000001 of capacity
<memory:100, vCores:1> on host storyacid-lm:55407, which currently has 1
containers, <memory:100, vCores:1> used and <memory:8092, vCores:15>
available
I am requesting containers with 100 mb mem and 1 core. If I know more
about how is the capacity if calculated per node , or how the allocation is
done , then it would be useful.
Thanks for the help!
Rahul
On Fri, Sep 6, 2013 at 10:31 AM, Devaraj k <de...@huawei.com> wrote:
> Hi Rahul,****
>
> ** **
>
> Could you tell me, what is the version you are using?****
>
> ****
>
> **· **If you want a container, you need to issue 3 resource
> requests (1-node local, 1-rack local and 1-Any(*) ). If you are using
> 2.1.0-beta<https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=ASC&sorter/field=priority&pid=12313722&customfield_12310320=12324029>or later versions, you can set the Relax Locality flag to false for getting
> only on the specified host.****
>
> Can you also share the code how you are requesting for containers…so that
> we can help you better..****
>
> ** **
>
> Thanks****
>
> Devaraj k****
>
> ** **
>
> *From:* Rahul Bhattacharjee [mailto:rahul.rec.dgp@gmail.com]
> *Sent:* 06 September 2013 09:43
> *To:* user@hadoop.apache.org
> *Subject:* Re: Question related to resource allocation in Yarn!****
>
> ** **
>
> I could progress a bit on this.
>
> I was not setting responseId while asking for containers.****
>
> Still I have one question as why I am only been allocated two containers
> whereas node manager can run more containers.
>
> Response while registering the application master -
> AM registration response minimumCapability {, memory: 1024, virtual_cores:
> 1, }, maximumCapability {, memory: 8192, virtual_cores: 32, }, ****
>
> Thanks,****
>
> Rahul****
>
> ** **
>
> On Thu, Sep 5, 2013 at 8:33 PM, Rahul Bhattacharjee <
> rahul.rec.dgp@gmail.com> wrote:****
>
> Hi,****
>
> I am trying to make a small poc on top of yarn.****
>
> Within the launched application master , I am trying to request for 50
> containers and launch a same task on those allocated containers.****
>
> My config : AM registration response minimumCapability {, memory: 1024,
> virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32,
> }, ****
>
> 1) I am asking for 1G mem and 1 core container to the RM. Ideally the RM
> should return me 6 - 7 containers , but the response always returns with
> only 2 containers. ****
>
> Why is that ?****
>
> 2) So , when in the first ask 2 containers are returned , then I again
> required the RM for 50 - 2 = 48 containers. I keep getting 0 containers ,
> even if the previously started containers have finished.****
>
> Why is that ?****
>
> Any link explaining the allocate request of RM would be very helpful.****
>
> ** **
>
> Thanks,
> Rahul****
>
> ** **
>
RE: Question related to resource allocation in Yarn!
Posted by Devaraj k <de...@huawei.com>.
Hi Rahul,
Could you tell me, what is the version you are using?
· If you want a container, you need to issue 3 resource requests (1-node local, 1-rack local and 1-Any(*) ). If you are using 2.1.0-beta<https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=ASC&sorter/field=priority&pid=12313722&customfield_12310320=12324029> or later versions, you can set the Relax Locality flag to false for getting only on the specified host.
Can you also share the code how you are requesting for containers…so that we can help you better..
Thanks
Devaraj k
From: Rahul Bhattacharjee [mailto:rahul.rec.dgp@gmail.com]
Sent: 06 September 2013 09:43
To: user@hadoop.apache.org
Subject: Re: Question related to resource allocation in Yarn!
I could progress a bit on this.
I was not setting responseId while asking for containers.
Still I have one question as why I am only been allocated two containers whereas node manager can run more containers.
Response while registering the application master -
AM registration response minimumCapability {, memory: 1024, virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32, },
Thanks,
Rahul
On Thu, Sep 5, 2013 at 8:33 PM, Rahul Bhattacharjee <ra...@gmail.com>> wrote:
Hi,
I am trying to make a small poc on top of yarn.
Within the launched application master , I am trying to request for 50 containers and launch a same task on those allocated containers.
My config : AM registration response minimumCapability {, memory: 1024, virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32, },
1) I am asking for 1G mem and 1 core container to the RM. Ideally the RM should return me 6 - 7 containers , but the response always returns with only 2 containers.
Why is that ?
2) So , when in the first ask 2 containers are returned , then I again required the RM for 50 - 2 = 48 containers. I keep getting 0 containers , even if the previously started containers have finished.
Why is that ?
Any link explaining the allocate request of RM would be very helpful.
Thanks,
Rahul
RE: Question related to resource allocation in Yarn!
Posted by Devaraj k <de...@huawei.com>.
Hi Rahul,
Could you tell me, what is the version you are using?
· If you want a container, you need to issue 3 resource requests (1-node local, 1-rack local and 1-Any(*) ). If you are using 2.1.0-beta<https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=ASC&sorter/field=priority&pid=12313722&customfield_12310320=12324029> or later versions, you can set the Relax Locality flag to false for getting only on the specified host.
Can you also share the code how you are requesting for containers…so that we can help you better..
Thanks
Devaraj k
From: Rahul Bhattacharjee [mailto:rahul.rec.dgp@gmail.com]
Sent: 06 September 2013 09:43
To: user@hadoop.apache.org
Subject: Re: Question related to resource allocation in Yarn!
I could progress a bit on this.
I was not setting responseId while asking for containers.
Still I have one question as why I am only been allocated two containers whereas node manager can run more containers.
Response while registering the application master -
AM registration response minimumCapability {, memory: 1024, virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32, },
Thanks,
Rahul
On Thu, Sep 5, 2013 at 8:33 PM, Rahul Bhattacharjee <ra...@gmail.com>> wrote:
Hi,
I am trying to make a small poc on top of yarn.
Within the launched application master , I am trying to request for 50 containers and launch a same task on those allocated containers.
My config : AM registration response minimumCapability {, memory: 1024, virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32, },
1) I am asking for 1G mem and 1 core container to the RM. Ideally the RM should return me 6 - 7 containers , but the response always returns with only 2 containers.
Why is that ?
2) So , when in the first ask 2 containers are returned , then I again required the RM for 50 - 2 = 48 containers. I keep getting 0 containers , even if the previously started containers have finished.
Why is that ?
Any link explaining the allocate request of RM would be very helpful.
Thanks,
Rahul
RE: Question related to resource allocation in Yarn!
Posted by Devaraj k <de...@huawei.com>.
Hi Rahul,
Could you tell me, what is the version you are using?
· If you want a container, you need to issue 3 resource requests (1-node local, 1-rack local and 1-Any(*) ). If you are using 2.1.0-beta<https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=ASC&sorter/field=priority&pid=12313722&customfield_12310320=12324029> or later versions, you can set the Relax Locality flag to false for getting only on the specified host.
Can you also share the code how you are requesting for containers…so that we can help you better..
Thanks
Devaraj k
From: Rahul Bhattacharjee [mailto:rahul.rec.dgp@gmail.com]
Sent: 06 September 2013 09:43
To: user@hadoop.apache.org
Subject: Re: Question related to resource allocation in Yarn!
I could progress a bit on this.
I was not setting responseId while asking for containers.
Still I have one question as why I am only been allocated two containers whereas node manager can run more containers.
Response while registering the application master -
AM registration response minimumCapability {, memory: 1024, virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32, },
Thanks,
Rahul
On Thu, Sep 5, 2013 at 8:33 PM, Rahul Bhattacharjee <ra...@gmail.com>> wrote:
Hi,
I am trying to make a small poc on top of yarn.
Within the launched application master , I am trying to request for 50 containers and launch a same task on those allocated containers.
My config : AM registration response minimumCapability {, memory: 1024, virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32, },
1) I am asking for 1G mem and 1 core container to the RM. Ideally the RM should return me 6 - 7 containers , but the response always returns with only 2 containers.
Why is that ?
2) So , when in the first ask 2 containers are returned , then I again required the RM for 50 - 2 = 48 containers. I keep getting 0 containers , even if the previously started containers have finished.
Why is that ?
Any link explaining the allocate request of RM would be very helpful.
Thanks,
Rahul
RE: Question related to resource allocation in Yarn!
Posted by Devaraj k <de...@huawei.com>.
Hi Rahul,
Could you tell me, what is the version you are using?
· If you want a container, you need to issue 3 resource requests (1-node local, 1-rack local and 1-Any(*) ). If you are using 2.1.0-beta<https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=ASC&sorter/field=priority&pid=12313722&customfield_12310320=12324029> or later versions, you can set the Relax Locality flag to false for getting only on the specified host.
Can you also share the code how you are requesting for containers…so that we can help you better..
Thanks
Devaraj k
From: Rahul Bhattacharjee [mailto:rahul.rec.dgp@gmail.com]
Sent: 06 September 2013 09:43
To: user@hadoop.apache.org
Subject: Re: Question related to resource allocation in Yarn!
I could progress a bit on this.
I was not setting responseId while asking for containers.
Still I have one question as why I am only been allocated two containers whereas node manager can run more containers.
Response while registering the application master -
AM registration response minimumCapability {, memory: 1024, virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32, },
Thanks,
Rahul
On Thu, Sep 5, 2013 at 8:33 PM, Rahul Bhattacharjee <ra...@gmail.com>> wrote:
Hi,
I am trying to make a small poc on top of yarn.
Within the launched application master , I am trying to request for 50 containers and launch a same task on those allocated containers.
My config : AM registration response minimumCapability {, memory: 1024, virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32, },
1) I am asking for 1G mem and 1 core container to the RM. Ideally the RM should return me 6 - 7 containers , but the response always returns with only 2 containers.
Why is that ?
2) So , when in the first ask 2 containers are returned , then I again required the RM for 50 - 2 = 48 containers. I keep getting 0 containers , even if the previously started containers have finished.
Why is that ?
Any link explaining the allocate request of RM would be very helpful.
Thanks,
Rahul
Re: Question related to resource allocation in Yarn!
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
I could progress a bit on this.
I was not setting responseId while asking for containers.
Still I have one question as why I am only been allocated two containers
whereas node manager can run more containers.
Response while registering the application master -
AM registration response minimumCapability {, memory: 1024, virtual_cores:
1, }, maximumCapability {, memory: 8192, virtual_cores: 32, },
Thanks,
Rahul
On Thu, Sep 5, 2013 at 8:33 PM, Rahul Bhattacharjee <rahul.rec.dgp@gmail.com
> wrote:
> Hi,
>
> I am trying to make a small poc on top of yarn.
>
> Within the launched application master , I am trying to request for 50
> containers and launch a same task on those allocated containers.
>
> My config : AM registration response minimumCapability {, memory: 1024,
> virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32,
> },
>
> 1) I am asking for 1G mem and 1 core container to the RM. Ideally the RM
> should return me 6 - 7 containers , but the response always returns with
> only 2 containers.
>
> Why is that ?
>
> 2) So , when in the first ask 2 containers are returned , then I again
> required the RM for 50 - 2 = 48 containers. I keep getting 0 containers ,
> even if the previously started containers have finished.
>
> Why is that ?
>
> Any link explaining the allocate request of RM would be very helpful.
>
> Thanks,
> Rahul
>
Re: Question related to resource allocation in Yarn!
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
I could progress a bit on this.
I was not setting responseId while asking for containers.
Still I have one question as why I am only been allocated two containers
whereas node manager can run more containers.
Response while registering the application master -
AM registration response minimumCapability {, memory: 1024, virtual_cores:
1, }, maximumCapability {, memory: 8192, virtual_cores: 32, },
Thanks,
Rahul
On Thu, Sep 5, 2013 at 8:33 PM, Rahul Bhattacharjee <rahul.rec.dgp@gmail.com
> wrote:
> Hi,
>
> I am trying to make a small poc on top of yarn.
>
> Within the launched application master , I am trying to request for 50
> containers and launch a same task on those allocated containers.
>
> My config : AM registration response minimumCapability {, memory: 1024,
> virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32,
> },
>
> 1) I am asking for 1G mem and 1 core container to the RM. Ideally the RM
> should return me 6 - 7 containers , but the response always returns with
> only 2 containers.
>
> Why is that ?
>
> 2) So , when in the first ask 2 containers are returned , then I again
> required the RM for 50 - 2 = 48 containers. I keep getting 0 containers ,
> even if the previously started containers have finished.
>
> Why is that ?
>
> Any link explaining the allocate request of RM would be very helpful.
>
> Thanks,
> Rahul
>
Re: Question related to resource allocation in Yarn!
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
I could progress a bit on this.
I was not setting responseId while asking for containers.
Still I have one question as why I am only been allocated two containers
whereas node manager can run more containers.
Response while registering the application master -
AM registration response minimumCapability {, memory: 1024, virtual_cores:
1, }, maximumCapability {, memory: 8192, virtual_cores: 32, },
Thanks,
Rahul
On Thu, Sep 5, 2013 at 8:33 PM, Rahul Bhattacharjee <rahul.rec.dgp@gmail.com
> wrote:
> Hi,
>
> I am trying to make a small poc on top of yarn.
>
> Within the launched application master , I am trying to request for 50
> containers and launch a same task on those allocated containers.
>
> My config : AM registration response minimumCapability {, memory: 1024,
> virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32,
> },
>
> 1) I am asking for 1G mem and 1 core container to the RM. Ideally the RM
> should return me 6 - 7 containers , but the response always returns with
> only 2 containers.
>
> Why is that ?
>
> 2) So , when in the first ask 2 containers are returned , then I again
> required the RM for 50 - 2 = 48 containers. I keep getting 0 containers ,
> even if the previously started containers have finished.
>
> Why is that ?
>
> Any link explaining the allocate request of RM would be very helpful.
>
> Thanks,
> Rahul
>
Re: Question related to resource allocation in Yarn!
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
I could progress a bit on this.
I was not setting responseId while asking for containers.
Still I have one question as why I am only been allocated two containers
whereas node manager can run more containers.
Response while registering the application master -
AM registration response minimumCapability {, memory: 1024, virtual_cores:
1, }, maximumCapability {, memory: 8192, virtual_cores: 32, },
Thanks,
Rahul
On Thu, Sep 5, 2013 at 8:33 PM, Rahul Bhattacharjee <rahul.rec.dgp@gmail.com
> wrote:
> Hi,
>
> I am trying to make a small poc on top of yarn.
>
> Within the launched application master , I am trying to request for 50
> containers and launch a same task on those allocated containers.
>
> My config : AM registration response minimumCapability {, memory: 1024,
> virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32,
> },
>
> 1) I am asking for 1G mem and 1 core container to the RM. Ideally the RM
> should return me 6 - 7 containers , but the response always returns with
> only 2 containers.
>
> Why is that ?
>
> 2) So , when in the first ask 2 containers are returned , then I again
> required the RM for 50 - 2 = 48 containers. I keep getting 0 containers ,
> even if the previously started containers have finished.
>
> Why is that ?
>
> Any link explaining the allocate request of RM would be very helpful.
>
> Thanks,
> Rahul
>