You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Bill Q <bi...@gmail.com> on 2013/12/09 05:52:13 UTC
Issues with ResourceManager scheduling functions
Hi,
I recently implemented an application using YARN. Everything works fine,
except for the illogical scheduling behavior of the ResourceManger, which
might be caused by the heterogeneity of the cluster.
I have two types of computing node in the cluster. 10 nodes with 128G
memory and 10 nodes with 64G memory. In the application, I create two types
of container. Type A container would ask for 50G of memory and Type B
container would ask for 100G of memory.
The problem is that if I created 50 Type B containers (100G), and 50 Type A
containers (50G), Only Type B container (100G) are running. Instead of
running the Type A container (50G), all the nodes with 64G memory are
idling. And only the nodes with 128G memory are running Type B container
(100G). This doesn't make too much sense to me.
If I only created Type A containers (50G), all the nodes would be able run
the containers. The 128G node would run two containers simultaneously. So,
the problem seems to be connected to the different types of containers.
Does anybody have any suggestions to fix this issue?
Many thanks.
Bill
Re: Issues with ResourceManager scheduling functions
Posted by Sandy Ryza <sa...@cloudera.com>.
Try requesting all your 100GB containers with priority 1 and your 50 GB
containers with priority 2.
-Sandy
On Mon, Dec 9, 2013 at 12:59 AM, Bill Q <bi...@gmail.com> wrote:
> Thanks a lot.
>
> So, in order for the two types of containers mentioned above to work, what
> priorities should I assign to them? Can you give an example?
>
> Many thanks.
>
>
> Bill
>
>
> On Mon, Dec 9, 2013 at 3:30 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> YARN currently is unable to handle requests with different resource
>> requirements at the same priority (YARN-314). Using different priorities
>> would likely solve the issue. I believe it would also be safest to avoid
>> using the priority 0 because this is used for the application master.
>>
>> -Sandy
>>
>>
>> On Mon, Dec 9, 2013 at 12:23 AM, Bill Q <bi...@gmail.com> wrote:
>>
>>> No, all default 0. Should I use different priorities?
>>>
>>> Many thanks.
>>>
>>>
>>> Bill
>>>
>>>
>>> On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>>>
>>>> Are you requesting the different container types at different
>>>> priorities?
>>>>
>>>> -Sandy
>>>>
>>>>
>>>> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> I recently implemented an application using YARN. Everything works
>>>>> fine, except for the illogical scheduling behavior of the ResourceManger,
>>>>> which might be caused by the heterogeneity of the cluster.
>>>>>
>>>>> I have two types of computing node in the cluster. 10 nodes with 128G
>>>>> memory and 10 nodes with 64G memory. In the application, I create two types
>>>>> of container. Type A container would ask for 50G of memory and Type B
>>>>> container would ask for 100G of memory.
>>>>>
>>>>> The problem is that if I created 50 Type B containers (100G), and 50
>>>>> Type A containers (50G), Only Type B container (100G) are running. Instead
>>>>> of running the Type A container (50G), all the nodes with 64G memory are
>>>>> idling. And only the nodes with 128G memory are running Type B container
>>>>> (100G). This doesn't make too much sense to me.
>>>>>
>>>>> If I only created Type A containers (50G), all the nodes would be able
>>>>> run the containers. The 128G node would run two containers simultaneously.
>>>>> So, the problem seems to be connected to the different types of containers.
>>>>>
>>>>> Does anybody have any suggestions to fix this issue?
>>>>>
>>>>> Many thanks.
>>>>>
>>>>>
>>>>> Bill
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Issues with ResourceManager scheduling functions
Posted by Sandy Ryza <sa...@cloudera.com>.
Try requesting all your 100GB containers with priority 1 and your 50 GB
containers with priority 2.
-Sandy
On Mon, Dec 9, 2013 at 12:59 AM, Bill Q <bi...@gmail.com> wrote:
> Thanks a lot.
>
> So, in order for the two types of containers mentioned above to work, what
> priorities should I assign to them? Can you give an example?
>
> Many thanks.
>
>
> Bill
>
>
> On Mon, Dec 9, 2013 at 3:30 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> YARN currently is unable to handle requests with different resource
>> requirements at the same priority (YARN-314). Using different priorities
>> would likely solve the issue. I believe it would also be safest to avoid
>> using the priority 0 because this is used for the application master.
>>
>> -Sandy
>>
>>
>> On Mon, Dec 9, 2013 at 12:23 AM, Bill Q <bi...@gmail.com> wrote:
>>
>>> No, all default 0. Should I use different priorities?
>>>
>>> Many thanks.
>>>
>>>
>>> Bill
>>>
>>>
>>> On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>>>
>>>> Are you requesting the different container types at different
>>>> priorities?
>>>>
>>>> -Sandy
>>>>
>>>>
>>>> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> I recently implemented an application using YARN. Everything works
>>>>> fine, except for the illogical scheduling behavior of the ResourceManger,
>>>>> which might be caused by the heterogeneity of the cluster.
>>>>>
>>>>> I have two types of computing node in the cluster. 10 nodes with 128G
>>>>> memory and 10 nodes with 64G memory. In the application, I create two types
>>>>> of container. Type A container would ask for 50G of memory and Type B
>>>>> container would ask for 100G of memory.
>>>>>
>>>>> The problem is that if I created 50 Type B containers (100G), and 50
>>>>> Type A containers (50G), Only Type B container (100G) are running. Instead
>>>>> of running the Type A container (50G), all the nodes with 64G memory are
>>>>> idling. And only the nodes with 128G memory are running Type B container
>>>>> (100G). This doesn't make too much sense to me.
>>>>>
>>>>> If I only created Type A containers (50G), all the nodes would be able
>>>>> run the containers. The 128G node would run two containers simultaneously.
>>>>> So, the problem seems to be connected to the different types of containers.
>>>>>
>>>>> Does anybody have any suggestions to fix this issue?
>>>>>
>>>>> Many thanks.
>>>>>
>>>>>
>>>>> Bill
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Issues with ResourceManager scheduling functions
Posted by Sandy Ryza <sa...@cloudera.com>.
Try requesting all your 100GB containers with priority 1 and your 50 GB
containers with priority 2.
-Sandy
On Mon, Dec 9, 2013 at 12:59 AM, Bill Q <bi...@gmail.com> wrote:
> Thanks a lot.
>
> So, in order for the two types of containers mentioned above to work, what
> priorities should I assign to them? Can you give an example?
>
> Many thanks.
>
>
> Bill
>
>
> On Mon, Dec 9, 2013 at 3:30 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> YARN currently is unable to handle requests with different resource
>> requirements at the same priority (YARN-314). Using different priorities
>> would likely solve the issue. I believe it would also be safest to avoid
>> using the priority 0 because this is used for the application master.
>>
>> -Sandy
>>
>>
>> On Mon, Dec 9, 2013 at 12:23 AM, Bill Q <bi...@gmail.com> wrote:
>>
>>> No, all default 0. Should I use different priorities?
>>>
>>> Many thanks.
>>>
>>>
>>> Bill
>>>
>>>
>>> On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>>>
>>>> Are you requesting the different container types at different
>>>> priorities?
>>>>
>>>> -Sandy
>>>>
>>>>
>>>> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> I recently implemented an application using YARN. Everything works
>>>>> fine, except for the illogical scheduling behavior of the ResourceManger,
>>>>> which might be caused by the heterogeneity of the cluster.
>>>>>
>>>>> I have two types of computing node in the cluster. 10 nodes with 128G
>>>>> memory and 10 nodes with 64G memory. In the application, I create two types
>>>>> of container. Type A container would ask for 50G of memory and Type B
>>>>> container would ask for 100G of memory.
>>>>>
>>>>> The problem is that if I created 50 Type B containers (100G), and 50
>>>>> Type A containers (50G), Only Type B container (100G) are running. Instead
>>>>> of running the Type A container (50G), all the nodes with 64G memory are
>>>>> idling. And only the nodes with 128G memory are running Type B container
>>>>> (100G). This doesn't make too much sense to me.
>>>>>
>>>>> If I only created Type A containers (50G), all the nodes would be able
>>>>> run the containers. The 128G node would run two containers simultaneously.
>>>>> So, the problem seems to be connected to the different types of containers.
>>>>>
>>>>> Does anybody have any suggestions to fix this issue?
>>>>>
>>>>> Many thanks.
>>>>>
>>>>>
>>>>> Bill
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Issues with ResourceManager scheduling functions
Posted by Sandy Ryza <sa...@cloudera.com>.
Try requesting all your 100GB containers with priority 1 and your 50 GB
containers with priority 2.
-Sandy
On Mon, Dec 9, 2013 at 12:59 AM, Bill Q <bi...@gmail.com> wrote:
> Thanks a lot.
>
> So, in order for the two types of containers mentioned above to work, what
> priorities should I assign to them? Can you give an example?
>
> Many thanks.
>
>
> Bill
>
>
> On Mon, Dec 9, 2013 at 3:30 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> YARN currently is unable to handle requests with different resource
>> requirements at the same priority (YARN-314). Using different priorities
>> would likely solve the issue. I believe it would also be safest to avoid
>> using the priority 0 because this is used for the application master.
>>
>> -Sandy
>>
>>
>> On Mon, Dec 9, 2013 at 12:23 AM, Bill Q <bi...@gmail.com> wrote:
>>
>>> No, all default 0. Should I use different priorities?
>>>
>>> Many thanks.
>>>
>>>
>>> Bill
>>>
>>>
>>> On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>>>
>>>> Are you requesting the different container types at different
>>>> priorities?
>>>>
>>>> -Sandy
>>>>
>>>>
>>>> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> I recently implemented an application using YARN. Everything works
>>>>> fine, except for the illogical scheduling behavior of the ResourceManger,
>>>>> which might be caused by the heterogeneity of the cluster.
>>>>>
>>>>> I have two types of computing node in the cluster. 10 nodes with 128G
>>>>> memory and 10 nodes with 64G memory. In the application, I create two types
>>>>> of container. Type A container would ask for 50G of memory and Type B
>>>>> container would ask for 100G of memory.
>>>>>
>>>>> The problem is that if I created 50 Type B containers (100G), and 50
>>>>> Type A containers (50G), Only Type B container (100G) are running. Instead
>>>>> of running the Type A container (50G), all the nodes with 64G memory are
>>>>> idling. And only the nodes with 128G memory are running Type B container
>>>>> (100G). This doesn't make too much sense to me.
>>>>>
>>>>> If I only created Type A containers (50G), all the nodes would be able
>>>>> run the containers. The 128G node would run two containers simultaneously.
>>>>> So, the problem seems to be connected to the different types of containers.
>>>>>
>>>>> Does anybody have any suggestions to fix this issue?
>>>>>
>>>>> Many thanks.
>>>>>
>>>>>
>>>>> Bill
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Issues with ResourceManager scheduling functions
Posted by Bill Q <bi...@gmail.com>.
Thanks a lot.
So, in order for the two types of containers mentioned above to work, what
priorities should I assign to them? Can you give an example?
Many thanks.
Bill
On Mon, Dec 9, 2013 at 3:30 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> YARN currently is unable to handle requests with different resource
> requirements at the same priority (YARN-314). Using different priorities
> would likely solve the issue. I believe it would also be safest to avoid
> using the priority 0 because this is used for the application master.
>
> -Sandy
>
>
> On Mon, Dec 9, 2013 at 12:23 AM, Bill Q <bi...@gmail.com> wrote:
>
>> No, all default 0. Should I use different priorities?
>>
>> Many thanks.
>>
>>
>> Bill
>>
>>
>> On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>>
>>> Are you requesting the different container types at different priorities?
>>>
>>> -Sandy
>>>
>>>
>>> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I recently implemented an application using YARN. Everything works
>>>> fine, except for the illogical scheduling behavior of the ResourceManger,
>>>> which might be caused by the heterogeneity of the cluster.
>>>>
>>>> I have two types of computing node in the cluster. 10 nodes with 128G
>>>> memory and 10 nodes with 64G memory. In the application, I create two types
>>>> of container. Type A container would ask for 50G of memory and Type B
>>>> container would ask for 100G of memory.
>>>>
>>>> The problem is that if I created 50 Type B containers (100G), and 50
>>>> Type A containers (50G), Only Type B container (100G) are running. Instead
>>>> of running the Type A container (50G), all the nodes with 64G memory are
>>>> idling. And only the nodes with 128G memory are running Type B container
>>>> (100G). This doesn't make too much sense to me.
>>>>
>>>> If I only created Type A containers (50G), all the nodes would be able
>>>> run the containers. The 128G node would run two containers simultaneously.
>>>> So, the problem seems to be connected to the different types of containers.
>>>>
>>>> Does anybody have any suggestions to fix this issue?
>>>>
>>>> Many thanks.
>>>>
>>>>
>>>> Bill
>>>>
>>>
>>>
>>
>
Re: Issues with ResourceManager scheduling functions
Posted by Bill Q <bi...@gmail.com>.
Thanks a lot.
So, in order for the two types of containers mentioned above to work, what
priorities should I assign to them? Can you give an example?
Many thanks.
Bill
On Mon, Dec 9, 2013 at 3:30 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> YARN currently is unable to handle requests with different resource
> requirements at the same priority (YARN-314). Using different priorities
> would likely solve the issue. I believe it would also be safest to avoid
> using the priority 0 because this is used for the application master.
>
> -Sandy
>
>
> On Mon, Dec 9, 2013 at 12:23 AM, Bill Q <bi...@gmail.com> wrote:
>
>> No, all default 0. Should I use different priorities?
>>
>> Many thanks.
>>
>>
>> Bill
>>
>>
>> On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>>
>>> Are you requesting the different container types at different priorities?
>>>
>>> -Sandy
>>>
>>>
>>> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I recently implemented an application using YARN. Everything works
>>>> fine, except for the illogical scheduling behavior of the ResourceManger,
>>>> which might be caused by the heterogeneity of the cluster.
>>>>
>>>> I have two types of computing node in the cluster. 10 nodes with 128G
>>>> memory and 10 nodes with 64G memory. In the application, I create two types
>>>> of container. Type A container would ask for 50G of memory and Type B
>>>> container would ask for 100G of memory.
>>>>
>>>> The problem is that if I created 50 Type B containers (100G), and 50
>>>> Type A containers (50G), Only Type B container (100G) are running. Instead
>>>> of running the Type A container (50G), all the nodes with 64G memory are
>>>> idling. And only the nodes with 128G memory are running Type B container
>>>> (100G). This doesn't make too much sense to me.
>>>>
>>>> If I only created Type A containers (50G), all the nodes would be able
>>>> run the containers. The 128G node would run two containers simultaneously.
>>>> So, the problem seems to be connected to the different types of containers.
>>>>
>>>> Does anybody have any suggestions to fix this issue?
>>>>
>>>> Many thanks.
>>>>
>>>>
>>>> Bill
>>>>
>>>
>>>
>>
>
Re: Issues with ResourceManager scheduling functions
Posted by Bill Q <bi...@gmail.com>.
Thanks a lot.
So, in order for the two types of containers mentioned above to work, what
priorities should I assign to them? Can you give an example?
Many thanks.
Bill
On Mon, Dec 9, 2013 at 3:30 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> YARN currently is unable to handle requests with different resource
> requirements at the same priority (YARN-314). Using different priorities
> would likely solve the issue. I believe it would also be safest to avoid
> using the priority 0 because this is used for the application master.
>
> -Sandy
>
>
> On Mon, Dec 9, 2013 at 12:23 AM, Bill Q <bi...@gmail.com> wrote:
>
>> No, all default 0. Should I use different priorities?
>>
>> Many thanks.
>>
>>
>> Bill
>>
>>
>> On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>>
>>> Are you requesting the different container types at different priorities?
>>>
>>> -Sandy
>>>
>>>
>>> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I recently implemented an application using YARN. Everything works
>>>> fine, except for the illogical scheduling behavior of the ResourceManger,
>>>> which might be caused by the heterogeneity of the cluster.
>>>>
>>>> I have two types of computing node in the cluster. 10 nodes with 128G
>>>> memory and 10 nodes with 64G memory. In the application, I create two types
>>>> of container. Type A container would ask for 50G of memory and Type B
>>>> container would ask for 100G of memory.
>>>>
>>>> The problem is that if I created 50 Type B containers (100G), and 50
>>>> Type A containers (50G), Only Type B container (100G) are running. Instead
>>>> of running the Type A container (50G), all the nodes with 64G memory are
>>>> idling. And only the nodes with 128G memory are running Type B container
>>>> (100G). This doesn't make too much sense to me.
>>>>
>>>> If I only created Type A containers (50G), all the nodes would be able
>>>> run the containers. The 128G node would run two containers simultaneously.
>>>> So, the problem seems to be connected to the different types of containers.
>>>>
>>>> Does anybody have any suggestions to fix this issue?
>>>>
>>>> Many thanks.
>>>>
>>>>
>>>> Bill
>>>>
>>>
>>>
>>
>
Re: Issues with ResourceManager scheduling functions
Posted by Bill Q <bi...@gmail.com>.
Thanks a lot.
So, in order for the two types of containers mentioned above to work, what
priorities should I assign to them? Can you give an example?
Many thanks.
Bill
On Mon, Dec 9, 2013 at 3:30 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> YARN currently is unable to handle requests with different resource
> requirements at the same priority (YARN-314). Using different priorities
> would likely solve the issue. I believe it would also be safest to avoid
> using the priority 0 because this is used for the application master.
>
> -Sandy
>
>
> On Mon, Dec 9, 2013 at 12:23 AM, Bill Q <bi...@gmail.com> wrote:
>
>> No, all default 0. Should I use different priorities?
>>
>> Many thanks.
>>
>>
>> Bill
>>
>>
>> On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>>
>>> Are you requesting the different container types at different priorities?
>>>
>>> -Sandy
>>>
>>>
>>> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I recently implemented an application using YARN. Everything works
>>>> fine, except for the illogical scheduling behavior of the ResourceManger,
>>>> which might be caused by the heterogeneity of the cluster.
>>>>
>>>> I have two types of computing node in the cluster. 10 nodes with 128G
>>>> memory and 10 nodes with 64G memory. In the application, I create two types
>>>> of container. Type A container would ask for 50G of memory and Type B
>>>> container would ask for 100G of memory.
>>>>
>>>> The problem is that if I created 50 Type B containers (100G), and 50
>>>> Type A containers (50G), Only Type B container (100G) are running. Instead
>>>> of running the Type A container (50G), all the nodes with 64G memory are
>>>> idling. And only the nodes with 128G memory are running Type B container
>>>> (100G). This doesn't make too much sense to me.
>>>>
>>>> If I only created Type A containers (50G), all the nodes would be able
>>>> run the containers. The 128G node would run two containers simultaneously.
>>>> So, the problem seems to be connected to the different types of containers.
>>>>
>>>> Does anybody have any suggestions to fix this issue?
>>>>
>>>> Many thanks.
>>>>
>>>>
>>>> Bill
>>>>
>>>
>>>
>>
>
Re: Issues with ResourceManager scheduling functions
Posted by Sandy Ryza <sa...@cloudera.com>.
YARN currently is unable to handle requests with different resource
requirements at the same priority (YARN-314). Using different priorities
would likely solve the issue. I believe it would also be safest to avoid
using the priority 0 because this is used for the application master.
-Sandy
On Mon, Dec 9, 2013 at 12:23 AM, Bill Q <bi...@gmail.com> wrote:
> No, all default 0. Should I use different priorities?
>
> Many thanks.
>
>
> Bill
>
>
> On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Are you requesting the different container types at different priorities?
>>
>> -Sandy
>>
>>
>> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
>>
>>> Hi,
>>> I recently implemented an application using YARN. Everything works fine,
>>> except for the illogical scheduling behavior of the ResourceManger, which
>>> might be caused by the heterogeneity of the cluster.
>>>
>>> I have two types of computing node in the cluster. 10 nodes with 128G
>>> memory and 10 nodes with 64G memory. In the application, I create two types
>>> of container. Type A container would ask for 50G of memory and Type B
>>> container would ask for 100G of memory.
>>>
>>> The problem is that if I created 50 Type B containers (100G), and 50
>>> Type A containers (50G), Only Type B container (100G) are running. Instead
>>> of running the Type A container (50G), all the nodes with 64G memory are
>>> idling. And only the nodes with 128G memory are running Type B container
>>> (100G). This doesn't make too much sense to me.
>>>
>>> If I only created Type A containers (50G), all the nodes would be able
>>> run the containers. The 128G node would run two containers simultaneously.
>>> So, the problem seems to be connected to the different types of containers.
>>>
>>> Does anybody have any suggestions to fix this issue?
>>>
>>> Many thanks.
>>>
>>>
>>> Bill
>>>
>>
>>
>
Re: Issues with ResourceManager scheduling functions
Posted by Sandy Ryza <sa...@cloudera.com>.
YARN currently is unable to handle requests with different resource
requirements at the same priority (YARN-314). Using different priorities
would likely solve the issue. I believe it would also be safest to avoid
using the priority 0 because this is used for the application master.
-Sandy
On Mon, Dec 9, 2013 at 12:23 AM, Bill Q <bi...@gmail.com> wrote:
> No, all default 0. Should I use different priorities?
>
> Many thanks.
>
>
> Bill
>
>
> On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Are you requesting the different container types at different priorities?
>>
>> -Sandy
>>
>>
>> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
>>
>>> Hi,
>>> I recently implemented an application using YARN. Everything works fine,
>>> except for the illogical scheduling behavior of the ResourceManger, which
>>> might be caused by the heterogeneity of the cluster.
>>>
>>> I have two types of computing node in the cluster. 10 nodes with 128G
>>> memory and 10 nodes with 64G memory. In the application, I create two types
>>> of container. Type A container would ask for 50G of memory and Type B
>>> container would ask for 100G of memory.
>>>
>>> The problem is that if I created 50 Type B containers (100G), and 50
>>> Type A containers (50G), Only Type B container (100G) are running. Instead
>>> of running the Type A container (50G), all the nodes with 64G memory are
>>> idling. And only the nodes with 128G memory are running Type B container
>>> (100G). This doesn't make too much sense to me.
>>>
>>> If I only created Type A containers (50G), all the nodes would be able
>>> run the containers. The 128G node would run two containers simultaneously.
>>> So, the problem seems to be connected to the different types of containers.
>>>
>>> Does anybody have any suggestions to fix this issue?
>>>
>>> Many thanks.
>>>
>>>
>>> Bill
>>>
>>
>>
>
Re: Issues with ResourceManager scheduling functions
Posted by Sandy Ryza <sa...@cloudera.com>.
YARN currently is unable to handle requests with different resource
requirements at the same priority (YARN-314). Using different priorities
would likely solve the issue. I believe it would also be safest to avoid
using the priority 0 because this is used for the application master.
-Sandy
On Mon, Dec 9, 2013 at 12:23 AM, Bill Q <bi...@gmail.com> wrote:
> No, all default 0. Should I use different priorities?
>
> Many thanks.
>
>
> Bill
>
>
> On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Are you requesting the different container types at different priorities?
>>
>> -Sandy
>>
>>
>> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
>>
>>> Hi,
>>> I recently implemented an application using YARN. Everything works fine,
>>> except for the illogical scheduling behavior of the ResourceManger, which
>>> might be caused by the heterogeneity of the cluster.
>>>
>>> I have two types of computing node in the cluster. 10 nodes with 128G
>>> memory and 10 nodes with 64G memory. In the application, I create two types
>>> of container. Type A container would ask for 50G of memory and Type B
>>> container would ask for 100G of memory.
>>>
>>> The problem is that if I created 50 Type B containers (100G), and 50
>>> Type A containers (50G), Only Type B container (100G) are running. Instead
>>> of running the Type A container (50G), all the nodes with 64G memory are
>>> idling. And only the nodes with 128G memory are running Type B container
>>> (100G). This doesn't make too much sense to me.
>>>
>>> If I only created Type A containers (50G), all the nodes would be able
>>> run the containers. The 128G node would run two containers simultaneously.
>>> So, the problem seems to be connected to the different types of containers.
>>>
>>> Does anybody have any suggestions to fix this issue?
>>>
>>> Many thanks.
>>>
>>>
>>> Bill
>>>
>>
>>
>
Re: Issues with ResourceManager scheduling functions
Posted by Sandy Ryza <sa...@cloudera.com>.
YARN currently is unable to handle requests with different resource
requirements at the same priority (YARN-314). Using different priorities
would likely solve the issue. I believe it would also be safest to avoid
using the priority 0 because this is used for the application master.
-Sandy
On Mon, Dec 9, 2013 at 12:23 AM, Bill Q <bi...@gmail.com> wrote:
> No, all default 0. Should I use different priorities?
>
> Many thanks.
>
>
> Bill
>
>
> On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <sa...@cloudera.com>wrote:
>
>> Are you requesting the different container types at different priorities?
>>
>> -Sandy
>>
>>
>> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
>>
>>> Hi,
>>> I recently implemented an application using YARN. Everything works fine,
>>> except for the illogical scheduling behavior of the ResourceManger, which
>>> might be caused by the heterogeneity of the cluster.
>>>
>>> I have two types of computing node in the cluster. 10 nodes with 128G
>>> memory and 10 nodes with 64G memory. In the application, I create two types
>>> of container. Type A container would ask for 50G of memory and Type B
>>> container would ask for 100G of memory.
>>>
>>> The problem is that if I created 50 Type B containers (100G), and 50
>>> Type A containers (50G), Only Type B container (100G) are running. Instead
>>> of running the Type A container (50G), all the nodes with 64G memory are
>>> idling. And only the nodes with 128G memory are running Type B container
>>> (100G). This doesn't make too much sense to me.
>>>
>>> If I only created Type A containers (50G), all the nodes would be able
>>> run the containers. The 128G node would run two containers simultaneously.
>>> So, the problem seems to be connected to the different types of containers.
>>>
>>> Does anybody have any suggestions to fix this issue?
>>>
>>> Many thanks.
>>>
>>>
>>> Bill
>>>
>>
>>
>
Re: Issues with ResourceManager scheduling functions
Posted by Bill Q <bi...@gmail.com>.
No, all default 0. Should I use different priorities?
Many thanks.
Bill
On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> Are you requesting the different container types at different priorities?
>
> -Sandy
>
>
> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
>
>> Hi,
>> I recently implemented an application using YARN. Everything works fine,
>> except for the illogical scheduling behavior of the ResourceManger, which
>> might be caused by the heterogeneity of the cluster.
>>
>> I have two types of computing node in the cluster. 10 nodes with 128G
>> memory and 10 nodes with 64G memory. In the application, I create two types
>> of container. Type A container would ask for 50G of memory and Type B
>> container would ask for 100G of memory.
>>
>> The problem is that if I created 50 Type B containers (100G), and 50 Type
>> A containers (50G), Only Type B container (100G) are running. Instead of
>> running the Type A container (50G), all the nodes with 64G memory are
>> idling. And only the nodes with 128G memory are running Type B container
>> (100G). This doesn't make too much sense to me.
>>
>> If I only created Type A containers (50G), all the nodes would be able
>> run the containers. The 128G node would run two containers simultaneously.
>> So, the problem seems to be connected to the different types of containers.
>>
>> Does anybody have any suggestions to fix this issue?
>>
>> Many thanks.
>>
>>
>> Bill
>>
>
>
Re: Issues with ResourceManager scheduling functions
Posted by Bill Q <bi...@gmail.com>.
No, all default 0. Should I use different priorities?
Many thanks.
Bill
On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> Are you requesting the different container types at different priorities?
>
> -Sandy
>
>
> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
>
>> Hi,
>> I recently implemented an application using YARN. Everything works fine,
>> except for the illogical scheduling behavior of the ResourceManger, which
>> might be caused by the heterogeneity of the cluster.
>>
>> I have two types of computing node in the cluster. 10 nodes with 128G
>> memory and 10 nodes with 64G memory. In the application, I create two types
>> of container. Type A container would ask for 50G of memory and Type B
>> container would ask for 100G of memory.
>>
>> The problem is that if I created 50 Type B containers (100G), and 50 Type
>> A containers (50G), Only Type B container (100G) are running. Instead of
>> running the Type A container (50G), all the nodes with 64G memory are
>> idling. And only the nodes with 128G memory are running Type B container
>> (100G). This doesn't make too much sense to me.
>>
>> If I only created Type A containers (50G), all the nodes would be able
>> run the containers. The 128G node would run two containers simultaneously.
>> So, the problem seems to be connected to the different types of containers.
>>
>> Does anybody have any suggestions to fix this issue?
>>
>> Many thanks.
>>
>>
>> Bill
>>
>
>
Re: Issues with ResourceManager scheduling functions
Posted by Bill Q <bi...@gmail.com>.
No, all default 0. Should I use different priorities?
Many thanks.
Bill
On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> Are you requesting the different container types at different priorities?
>
> -Sandy
>
>
> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
>
>> Hi,
>> I recently implemented an application using YARN. Everything works fine,
>> except for the illogical scheduling behavior of the ResourceManger, which
>> might be caused by the heterogeneity of the cluster.
>>
>> I have two types of computing node in the cluster. 10 nodes with 128G
>> memory and 10 nodes with 64G memory. In the application, I create two types
>> of container. Type A container would ask for 50G of memory and Type B
>> container would ask for 100G of memory.
>>
>> The problem is that if I created 50 Type B containers (100G), and 50 Type
>> A containers (50G), Only Type B container (100G) are running. Instead of
>> running the Type A container (50G), all the nodes with 64G memory are
>> idling. And only the nodes with 128G memory are running Type B container
>> (100G). This doesn't make too much sense to me.
>>
>> If I only created Type A containers (50G), all the nodes would be able
>> run the containers. The 128G node would run two containers simultaneously.
>> So, the problem seems to be connected to the different types of containers.
>>
>> Does anybody have any suggestions to fix this issue?
>>
>> Many thanks.
>>
>>
>> Bill
>>
>
>
Re: Issues with ResourceManager scheduling functions
Posted by Bill Q <bi...@gmail.com>.
No, all default 0. Should I use different priorities?
Many thanks.
Bill
On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> Are you requesting the different container types at different priorities?
>
> -Sandy
>
>
> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
>
>> Hi,
>> I recently implemented an application using YARN. Everything works fine,
>> except for the illogical scheduling behavior of the ResourceManger, which
>> might be caused by the heterogeneity of the cluster.
>>
>> I have two types of computing node in the cluster. 10 nodes with 128G
>> memory and 10 nodes with 64G memory. In the application, I create two types
>> of container. Type A container would ask for 50G of memory and Type B
>> container would ask for 100G of memory.
>>
>> The problem is that if I created 50 Type B containers (100G), and 50 Type
>> A containers (50G), Only Type B container (100G) are running. Instead of
>> running the Type A container (50G), all the nodes with 64G memory are
>> idling. And only the nodes with 128G memory are running Type B container
>> (100G). This doesn't make too much sense to me.
>>
>> If I only created Type A containers (50G), all the nodes would be able
>> run the containers. The 128G node would run two containers simultaneously.
>> So, the problem seems to be connected to the different types of containers.
>>
>> Does anybody have any suggestions to fix this issue?
>>
>> Many thanks.
>>
>>
>> Bill
>>
>
>
Re: Issues with ResourceManager scheduling functions
Posted by Sandy Ryza <sa...@cloudera.com>.
Are you requesting the different container types at different priorities?
-Sandy
On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
> Hi,
> I recently implemented an application using YARN. Everything works fine,
> except for the illogical scheduling behavior of the ResourceManger, which
> might be caused by the heterogeneity of the cluster.
>
> I have two types of computing node in the cluster. 10 nodes with 128G
> memory and 10 nodes with 64G memory. In the application, I create two types
> of container. Type A container would ask for 50G of memory and Type B
> container would ask for 100G of memory.
>
> The problem is that if I created 50 Type B containers (100G), and 50 Type
> A containers (50G), Only Type B container (100G) are running. Instead of
> running the Type A container (50G), all the nodes with 64G memory are
> idling. And only the nodes with 128G memory are running Type B container
> (100G). This doesn't make too much sense to me.
>
> If I only created Type A containers (50G), all the nodes would be able run
> the containers. The 128G node would run two containers simultaneously. So,
> the problem seems to be connected to the different types of containers.
>
> Does anybody have any suggestions to fix this issue?
>
> Many thanks.
>
>
> Bill
>
Re: Issues with ResourceManager scheduling functions
Posted by Sandy Ryza <sa...@cloudera.com>.
Are you requesting the different container types at different priorities?
-Sandy
On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
> Hi,
> I recently implemented an application using YARN. Everything works fine,
> except for the illogical scheduling behavior of the ResourceManger, which
> might be caused by the heterogeneity of the cluster.
>
> I have two types of computing node in the cluster. 10 nodes with 128G
> memory and 10 nodes with 64G memory. In the application, I create two types
> of container. Type A container would ask for 50G of memory and Type B
> container would ask for 100G of memory.
>
> The problem is that if I created 50 Type B containers (100G), and 50 Type
> A containers (50G), Only Type B container (100G) are running. Instead of
> running the Type A container (50G), all the nodes with 64G memory are
> idling. And only the nodes with 128G memory are running Type B container
> (100G). This doesn't make too much sense to me.
>
> If I only created Type A containers (50G), all the nodes would be able run
> the containers. The 128G node would run two containers simultaneously. So,
> the problem seems to be connected to the different types of containers.
>
> Does anybody have any suggestions to fix this issue?
>
> Many thanks.
>
>
> Bill
>
Re: Issues with ResourceManager scheduling functions
Posted by Sandy Ryza <sa...@cloudera.com>.
Are you requesting the different container types at different priorities?
-Sandy
On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
> Hi,
> I recently implemented an application using YARN. Everything works fine,
> except for the illogical scheduling behavior of the ResourceManger, which
> might be caused by the heterogeneity of the cluster.
>
> I have two types of computing node in the cluster. 10 nodes with 128G
> memory and 10 nodes with 64G memory. In the application, I create two types
> of container. Type A container would ask for 50G of memory and Type B
> container would ask for 100G of memory.
>
> The problem is that if I created 50 Type B containers (100G), and 50 Type
> A containers (50G), Only Type B container (100G) are running. Instead of
> running the Type A container (50G), all the nodes with 64G memory are
> idling. And only the nodes with 128G memory are running Type B container
> (100G). This doesn't make too much sense to me.
>
> If I only created Type A containers (50G), all the nodes would be able run
> the containers. The 128G node would run two containers simultaneously. So,
> the problem seems to be connected to the different types of containers.
>
> Does anybody have any suggestions to fix this issue?
>
> Many thanks.
>
>
> Bill
>
Re: Issues with ResourceManager scheduling functions
Posted by Sandy Ryza <sa...@cloudera.com>.
Are you requesting the different container types at different priorities?
-Sandy
On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bi...@gmail.com> wrote:
> Hi,
> I recently implemented an application using YARN. Everything works fine,
> except for the illogical scheduling behavior of the ResourceManger, which
> might be caused by the heterogeneity of the cluster.
>
> I have two types of computing node in the cluster. 10 nodes with 128G
> memory and 10 nodes with 64G memory. In the application, I create two types
> of container. Type A container would ask for 50G of memory and Type B
> container would ask for 100G of memory.
>
> The problem is that if I created 50 Type B containers (100G), and 50 Type
> A containers (50G), Only Type B container (100G) are running. Instead of
> running the Type A container (50G), all the nodes with 64G memory are
> idling. And only the nodes with 128G memory are running Type B container
> (100G). This doesn't make too much sense to me.
>
> If I only created Type A containers (50G), all the nodes would be able run
> the containers. The 128G node would run two containers simultaneously. So,
> the problem seems to be connected to the different types of containers.
>
> Does anybody have any suggestions to fix this issue?
>
> Many thanks.
>
>
> Bill
>