You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Yuzhang Han <yu...@gmail.com> on 2013/06/14 22:05:07 UTC
Different container sizes for MR tasks, job does not complete
Hi,
I am using MapReduce on YARN. I want to make tasks of the same job run
in containers with different sizes. For example: Job1 = <Task1, ...,
Task8>. Task1 := 1280 MB; Task2 to 8 := 1024 MB.
To achieve this, I manually call
reqEvent.getCapability().setMemory(MEMORY_SIZE) in
RMContainerAllocator.java with MEMORY_SIZE = 1280 or 1024. The problem
is, if one task has different memory than the others, then the whole job
stops at 80% forever. When all the tasks have the same memory, the job
completes correctly and fast. Does some one has any idea?
Thanks a lot.
Yuzhang