Memory Manager in Hadoop MR

[Pedro Costa <ps...@gmail.com>]

Hi,

1 - Hadoop MR contains a TaskMemoryManagerThread class that is used to
manage memory usage of tasks running under a TaskTracker. Why Hadoop
MR needs a class to manage memory? Why it couldn't rely on the JVM, or
this class is here for another purpose?

2 - How the JT knows that a Map or Reduce Task finished? Is through
the heartbeat?

Thanks

-- 
Pedro

Re: Memory Manager in Hadoop MR

[Ted Yu <yu...@gmail.com>]

For 1, TMMT uses ProcessTree to check for task that is running beyond
memory-limits and kills it.

On Thu, Dec 9, 2010 at 3:05 AM, Pedro Costa <ps...@gmail.com> wrote:

> Hi,
>
> 1 - Hadoop MR contains a TaskMemoryManagerThread class that is used to
> manage memory usage of tasks running under a TaskTracker. Why Hadoop
> MR needs a class to manage memory? Why it couldn't rely on the JVM, or
> this class is here for another purpose?
>
> 2 - How the JT knows that a Map or Reduce Task finished? Is through
> the heartbeat?
>
> Thanks
>
> --
> Pedro
>

Re: Memory Manager in Hadoop MR

[Hemanth Yamijala <yh...@gmail.com>]

Hi,

On Thu, Dec 9, 2010 at 4:35 PM, Pedro Costa <ps...@gmail.com> wrote:
> Hi,
>
> 1 - Hadoop MR contains a TaskMemoryManagerThread class that is used to
> manage memory usage of tasks running under a TaskTracker. Why Hadoop
> MR needs a class to manage memory? Why it couldn't rely on the JVM, or
> this class is here for another purpose?
>

There are streaming and pipes map/reduce applications that launch
native processes from the map/reduce tasks that are outside the
control of the JVM. Indeed, even regular Java map/reduce programs
could fork/exec other programs. All of these processes could consume
memory that would not be accounted for if we relied only on the JVM to
get the memory usage. Hence a separate class that looks at the entire
process tree of the map/reduce task to account for memory consumed.

> 2 - How the JT knows that a Map or Reduce Task finished? Is through
> the heartbeat?
>

Yes.

> Thanks
>
> --
> Pedro
>

Re: Memory Manager in Hadoop MR

[Greg Roelofs <ro...@yahoo-inc.com>]

> 2 - How the JT knows that a Map or Reduce Task finished? Is through
> the heartbeat?

Exactly.  Tasks communicate with their TTs through the umbilical, and each
TT communicates with the JT via heartbeat (and heartbeat response).

Greg