You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Sandy Ryza <sa...@cloudera.com> on 2013/06/13 18:47:22 UTC
Re: Container size configuration
Hi Yuzhang,
Moving this question to the Hadoop user list.
Are you using MapReduce or writing your own YARN application? In
MapReduce, all maps must request the same amount of memory and all reduces
must request the same amount of memory. It would be trivial to do this in
your own YARN application.
-Sandy
On Thu, Jun 13, 2013 at 8:41 AM, Yuzhang Han <yu...@gmail.com>wrote:
> Hi,
>
> I am wondering if I can allocate different size of containers to the tasks
> in a job. For example: Job = <Task1, Task2, Task3>, Task1 = Task2 = 1024MB,
> Task3 = 2048MB. How can I achieve this? Many thanks.
>
> Yuzhang
>
Re: Container size configuration
Posted by Sandy Ryza <sa...@cloudera.com>.
You mean different memory sizes for different reducers? If you're willing
to mess with the MR Application Master code, you should be to do it there.
Check out RMContainerAllocator.java and RMContainerRequestor.java.
-Sandy
On Thu, Jun 13, 2013 at 9:56 AM, Yuzhang Han <yu...@gmail.com>wrote:
> Thank you Sandy.
>
> I am trying to do to some experiments with MapReduce on YARN, where some
> reducers gain larger processing tasks than the other reducers. I want to
> see that if I allocate more memory to the larger-task reducers, how the
> performance would be improved.
>
> Anyway, if I want to manually configure memory size for a particular
> reducer, how can I do it? In ApplicationMaster? Or this would not work at
> all?
>
> Thank you.
>
>
>
>
>
> On 6/13/2013 12:47 PM, Sandy Ryza wrote:
>
> Hi Yuzhang,
>
> Moving this question to the Hadoop user list.
>
> Are you using MapReduce or writing your own YARN application? In
> MapReduce, all maps must request the same amount of memory and all reduces
> must request the same amount of memory. It would be trivial to do this in
> your own YARN application.
>
> -Sandy
>
>
> On Thu, Jun 13, 2013 at 8:41 AM, Yuzhang Han <yu...@gmail.com>wrote:
>
>> Hi,
>>
>> I am wondering if I can allocate different size of containers to the
>> tasks in a job. For example: Job = <Task1, Task2, Task3>, Task1 = Task2 =
>> 1024MB, Task3 = 2048MB. How can I achieve this? Many thanks.
>>
>> Yuzhang
>>
>
>
>
Re: Container size configuration
Posted by Sandy Ryza <sa...@cloudera.com>.
You mean different memory sizes for different reducers? If you're willing
to mess with the MR Application Master code, you should be to do it there.
Check out RMContainerAllocator.java and RMContainerRequestor.java.
-Sandy
On Thu, Jun 13, 2013 at 9:56 AM, Yuzhang Han <yu...@gmail.com>wrote:
> Thank you Sandy.
>
> I am trying to do to some experiments with MapReduce on YARN, where some
> reducers gain larger processing tasks than the other reducers. I want to
> see that if I allocate more memory to the larger-task reducers, how the
> performance would be improved.
>
> Anyway, if I want to manually configure memory size for a particular
> reducer, how can I do it? In ApplicationMaster? Or this would not work at
> all?
>
> Thank you.
>
>
>
>
>
> On 6/13/2013 12:47 PM, Sandy Ryza wrote:
>
> Hi Yuzhang,
>
> Moving this question to the Hadoop user list.
>
> Are you using MapReduce or writing your own YARN application? In
> MapReduce, all maps must request the same amount of memory and all reduces
> must request the same amount of memory. It would be trivial to do this in
> your own YARN application.
>
> -Sandy
>
>
> On Thu, Jun 13, 2013 at 8:41 AM, Yuzhang Han <yu...@gmail.com>wrote:
>
>> Hi,
>>
>> I am wondering if I can allocate different size of containers to the
>> tasks in a job. For example: Job = <Task1, Task2, Task3>, Task1 = Task2 =
>> 1024MB, Task3 = 2048MB. How can I achieve this? Many thanks.
>>
>> Yuzhang
>>
>
>
>
Re: Container size configuration
Posted by Sandy Ryza <sa...@cloudera.com>.
You mean different memory sizes for different reducers? If you're willing
to mess with the MR Application Master code, you should be to do it there.
Check out RMContainerAllocator.java and RMContainerRequestor.java.
-Sandy
On Thu, Jun 13, 2013 at 9:56 AM, Yuzhang Han <yu...@gmail.com>wrote:
> Thank you Sandy.
>
> I am trying to do to some experiments with MapReduce on YARN, where some
> reducers gain larger processing tasks than the other reducers. I want to
> see that if I allocate more memory to the larger-task reducers, how the
> performance would be improved.
>
> Anyway, if I want to manually configure memory size for a particular
> reducer, how can I do it? In ApplicationMaster? Or this would not work at
> all?
>
> Thank you.
>
>
>
>
>
> On 6/13/2013 12:47 PM, Sandy Ryza wrote:
>
> Hi Yuzhang,
>
> Moving this question to the Hadoop user list.
>
> Are you using MapReduce or writing your own YARN application? In
> MapReduce, all maps must request the same amount of memory and all reduces
> must request the same amount of memory. It would be trivial to do this in
> your own YARN application.
>
> -Sandy
>
>
> On Thu, Jun 13, 2013 at 8:41 AM, Yuzhang Han <yu...@gmail.com>wrote:
>
>> Hi,
>>
>> I am wondering if I can allocate different size of containers to the
>> tasks in a job. For example: Job = <Task1, Task2, Task3>, Task1 = Task2 =
>> 1024MB, Task3 = 2048MB. How can I achieve this? Many thanks.
>>
>> Yuzhang
>>
>
>
>
Re: Container size configuration
Posted by Sandy Ryza <sa...@cloudera.com>.
You mean different memory sizes for different reducers? If you're willing
to mess with the MR Application Master code, you should be to do it there.
Check out RMContainerAllocator.java and RMContainerRequestor.java.
-Sandy
On Thu, Jun 13, 2013 at 9:56 AM, Yuzhang Han <yu...@gmail.com>wrote:
> Thank you Sandy.
>
> I am trying to do to some experiments with MapReduce on YARN, where some
> reducers gain larger processing tasks than the other reducers. I want to
> see that if I allocate more memory to the larger-task reducers, how the
> performance would be improved.
>
> Anyway, if I want to manually configure memory size for a particular
> reducer, how can I do it? In ApplicationMaster? Or this would not work at
> all?
>
> Thank you.
>
>
>
>
>
> On 6/13/2013 12:47 PM, Sandy Ryza wrote:
>
> Hi Yuzhang,
>
> Moving this question to the Hadoop user list.
>
> Are you using MapReduce or writing your own YARN application? In
> MapReduce, all maps must request the same amount of memory and all reduces
> must request the same amount of memory. It would be trivial to do this in
> your own YARN application.
>
> -Sandy
>
>
> On Thu, Jun 13, 2013 at 8:41 AM, Yuzhang Han <yu...@gmail.com>wrote:
>
>> Hi,
>>
>> I am wondering if I can allocate different size of containers to the
>> tasks in a job. For example: Job = <Task1, Task2, Task3>, Task1 = Task2 =
>> 1024MB, Task3 = 2048MB. How can I achieve this? Many thanks.
>>
>> Yuzhang
>>
>
>
>