You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by "Dvorkin-Contractor, Eugene (CORP)" <Eu...@ADP.com> on 2015/09/01 23:14:11 UTC
minimum requirements for hdfs on mesos
I am trying to run hdfs on mesos using mesosphere/hdfs repository.
I am getting errors
Info - o.a.m.hdfs.scheduler.HdfsScheduler - Offer does not have enough resources
What is the minimum requirement to run hdfs node or how can I adjust this.
Thanks
----------------------------------------------------------------------
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.
Re: minimum requirements for hdfs on mesos
Posted by "Dvorkin-Contractor, Eugene (CORP)" <Eu...@ADP.com>.
Thank you. I was able successfully install hdfs on mesos.
From: Ken Sipe <ke...@gmail.com>>
Reply-To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Date: Tuesday, September 1, 2015 at 5:38 PM
To: "user@mesos.apache.org<ma...@mesos.apache.org>" <us...@mesos.apache.org>>
Subject: Re: minimum requirements for hdfs on mesos
adding to Elizabeths comments check out: https://github.com/mesosphere/hdfs/blob/master/conf/mesos-site.xml#L92-L148
default settings are here: https://github.com/mesosphere/hdfs/blob/master/hdfs-commons/src/main/java/org/apache/mesos/hdfs/config/HdfsFrameworkConfig.java#L22-L38
ken
On Sep 1, 2015, at 4:33 PM, Elizabeth Lingg <el...@mesosphere.io>> wrote:
We recommend a minimum of 4 nodes with 4 CPU shares and 8GB of RAM available for the default settings. You can lower the settings by adjusting the configuration, but if you do so, you will not be able to store as much data.
Thanks,
Elizabeth
On Tue, Sep 1, 2015 at 2:14 PM, Dvorkin-Contractor, Eugene (CORP) <Eu...@adp.com>> wrote:
I am trying to run hdfs on mesos using mesosphere/hdfs repository.
I am getting errors
Info - o.a.m.hdfs.scheduler.HdfsScheduler - Offer does not have enough resources
What is the minimum requirement to run hdfs node or how can I adjust this.
Thanks
________________________________
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.
Re: minimum requirements for hdfs on mesos
Posted by Ken Sipe <ke...@gmail.com>.
adding to Elizabeths comments check out: https://github.com/mesosphere/hdfs/blob/master/conf/mesos-site.xml#L92-L148 <https://github.com/mesosphere/hdfs/blob/master/conf/mesos-site.xml#L92-L148>
default settings are here: https://github.com/mesosphere/hdfs/blob/master/hdfs-commons/src/main/java/org/apache/mesos/hdfs/config/HdfsFrameworkConfig.java#L22-L38 <https://github.com/mesosphere/hdfs/blob/master/hdfs-commons/src/main/java/org/apache/mesos/hdfs/config/HdfsFrameworkConfig.java#L22-L38>
ken
> On Sep 1, 2015, at 4:33 PM, Elizabeth Lingg <el...@mesosphere.io> wrote:
>
> We recommend a minimum of 4 nodes with 4 CPU shares and 8GB of RAM available for the default settings. You can lower the settings by adjusting the configuration, but if you do so, you will not be able to store as much data.
>
> Thanks,
> Elizabeth
>
>
>
> On Tue, Sep 1, 2015 at 2:14 PM, Dvorkin-Contractor, Eugene (CORP) <Eugene.Dvorkin-Contractor@adp.com <ma...@adp.com>> wrote:
> I am trying to run hdfs on mesos using mesosphere/hdfs repository.
> I am getting errors
>
> Info - o.a.m.hdfs.scheduler.HdfsScheduler - Offer does not have enough resources
>
> What is the minimum requirement to run hdfs node or how can I adjust this.
> Thanks
> This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.
>
Re: minimum requirements for hdfs on mesos
Posted by Elizabeth Lingg <el...@mesosphere.io>.
We recommend a minimum of 4 nodes with 4 CPU shares and 8GB of RAM
available for the default settings. You can lower the settings by adjusting
the configuration, but if you do so, you will not be able to store as much
data.
Thanks,
Elizabeth
On Tue, Sep 1, 2015 at 2:14 PM, Dvorkin-Contractor, Eugene (CORP) <
Eugene.Dvorkin-Contractor@adp.com> wrote:
> I am trying to run hdfs on mesos using mesosphere/hdfs repository.
> I am getting errors
>
> Info - o.a.m.hdfs.scheduler.HdfsScheduler - Offer does not have enough
> resources
>
> What is the minimum requirement to run hdfs node or how can I adjust this.
> Thanks
> ------------------------------
> This message and any attachments are intended only for the use of the
> addressee and may contain information that is privileged and confidential.
> If the reader of the message is not the intended recipient or an authorized
> representative of the intended recipient, you are hereby notified that any
> dissemination of this communication is strictly prohibited. If you have
> received this communication in error, notify the sender immediately by
> return email and delete the message and any attachments from your system.
>
Re: minimum requirements for hdfs on mesos
Posted by Gabriel Hartmann <ga...@mesosphere.io>.
hdfs/example-conf/mesosphere-dcos/mesos-site.xml has some example values
defined. For example:
<property>
<name>mesos.hdfs.namenode.cpus</name>
<value>0.25</value>
</property>
Furthermore you need at least 5 private slaves to satisfy the restrictions
on node collocation. In other words particular nodes are not allowed to
run on the same host. For example both NameNodes cannot run on the same
Host. I believe (I could be wrong) journal nodes may not run on the same
hosts as Datanodes etc. You can
consult hdfs/hdfs-scheduler/src/main/java/org/apache/mesos/hdfs/HdfsScheduler.java
to see the explicit circumstances under which resource offers are rejected.
-- Gabriel
On Tue, Sep 1, 2015 at 2:14 PM, Dvorkin-Contractor, Eugene (CORP) <
Eugene.Dvorkin-Contractor@adp.com> wrote:
> I am trying to run hdfs on mesos using mesosphere/hdfs repository.
> I am getting errors
>
> Info - o.a.m.hdfs.scheduler.HdfsScheduler - Offer does not have enough
> resources
>
> What is the minimum requirement to run hdfs node or how can I adjust this.
> Thanks
> ------------------------------
> This message and any attachments are intended only for the use of the
> addressee and may contain information that is privileged and confidential.
> If the reader of the message is not the intended recipient or an authorized
> representative of the intended recipient, you are hereby notified that any
> dissemination of this communication is strictly prohibited. If you have
> received this communication in error, notify the sender immediately by
> return email and delete the message and any attachments from your system.
>