Hadoop on Mesos Memory Configuration Question

[Ajit Jagdale <aj...@qualia-media.com>]

Hi all,

I'm new to Mesos and to using Hadoop over Mesos.  I've been trying to
determine if Mesos memory configurations are affecting the memory that I
allocate to Hadoop mappers and reducers (in Hadoop's mapped-site.xml
file).  When I set values to the mappers, something seems to interfere with
allocating that memory.

Cluster setup:
- 1 master node and 6 slave nodes
- There is no /etc/mesos-slave/resource file, so memory is configured by
Mesos.  My understanding of this is that since there are no explicit memory
settings on each slave node, Mesos is giving the asking application
(Hadoop) all of the available memory minus 1GB for running the OS.

But there still must be some mesos memory configuration somewhere, right?
Something that knows how much a slice of memory is.  I'm not sure if I know
where that is.

Any suggestions of how mesos' process of memory allocation could be
affecting how Hadoop affects memory allocation would be appreciated.

Thanks,
Ajit

Re: Hadoop on Mesos Memory Configuration Question

[Adam Bordelon <ad...@mesosphere.io>]

Mesos will take the resources from each slave (total-1GB) and offer that to
the various frameworks registered on your cluster. If Hadoop is the only
framework, it will get offered all of the resources. If there are other
frameworks running, Mesos will offer resources to the framework furthest
below its fair share, to try to maintain Dominant Resource Fairness (see
the DRF paper). It is up to the Hadoop framework scheduler to decide how
much of each offer to use to launch its tasks (TaskTrackers/NodeManagers).
Are you using Hadoop1 framework: https://github.com/mesos/hadoop
Or the Hadoop2/YARN framework: https://github.com/apache/incubator-myriad

On Thu, Oct 1, 2015 at 6:26 PM, Ajit Jagdale <aj...@qualia-media.com> wrote:

> Hi all,
>
> I'm new to Mesos and to using Hadoop over Mesos.  I've been trying to
> determine if Mesos memory configurations are affecting the memory that I
> allocate to Hadoop mappers and reducers (in Hadoop's mapped-site.xml
> file).  When I set values to the mappers, something seems to interfere with
> allocating that memory.
>
> Cluster setup:
> - 1 master node and 6 slave nodes
> - There is no /etc/mesos-slave/resource file, so memory is configured by
> Mesos.  My understanding of this is that since there are no explicit memory
> settings on each slave node, Mesos is giving the asking application
> (Hadoop) all of the available memory minus 1GB for running the OS.
>
> But there still must be some mesos memory configuration somewhere, right?
> Something that knows how much a slice of memory is.  I'm not sure if I know
> where that is.
>
> Any suggestions of how mesos' process of memory allocation could be
> affecting how Hadoop affects memory allocation would be appreciated.
>
> Thanks,
> Ajit
>